[jira] [Commented] (FLINK-12859) ClassNotFoundException during MapRFs init
[
https://issues.apache.org/jira/browse/FLINK-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864934#comment-16864934
]
Yun Tang commented on FLINK-12859:
--
Actually I cannot reproduce your problem locally and recent travis built seems
not meeting this problem. I think you could verify whether
{{commons-configuration}} jar package is completed and also located in your
class path when running tests.
> ClassNotFoundException during MapRFs init
> -
>
> Key: FLINK-12859
> URL: https://issues.apache.org/jira/browse/FLINK-12859
> Project: Flink
> Issue Type: Test
> Components: FileSystems
>Affects Versions: 1.8.0
>Reporter: Aitozi
>Priority: Minor
> Attachments: 20190616110004.jpg
>
>
> I was running _mvn clean package_ locally on master branch to pass all the
> test case, I ran into the _ClassNotFoundException_ as shown below, but I
> check the [flink-mapr-fs][pom.xml], it has not been changed for a long time,
> so is this always a failed test case?
> !image-2019-06-15-23-56-56-158.png!
> TestCase is: FileSystemAccessTest.testGetMapRFs()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] yanghua commented on a change in pull request #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure
yanghua commented on a change in pull request #8322: [FLINK-12364] Introduce a
CheckpointFailureManager to centralized manage checkpoint failure
URL: https://github.com/apache/flink/pull/8322#discussion_r294067081
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -1329,10 +1331,13 @@ private void discardCheckpoint(PendingCheckpoint
pendingCheckpoint, @Nullable Th
LOG.info("Discarding checkpoint {} of job {}.", checkpointId,
job, cause);
- if (cause == null || cause instanceof
CheckpointDeclineException) {
-
pendingCheckpoint.abort(CheckpointFailureReason.CHECKPOINT_DECLINED, cause);
+ if (cause == null) {
+ failPendingCheckpoint(pendingCheckpoint,
CheckpointFailureReason.CHECKPOINT_DECLINED);
+ } else if (cause instanceof CheckpointException) {
+ CheckpointException exception = (CheckpointException)
cause;
+ failPendingCheckpoint(pendingCheckpoint,
exception.getCheckpointFailureReason(), cause);
} else {
-
pendingCheckpoint.abort(CheckpointFailureReason.JOB_FAILURE, cause);
+ failPendingCheckpoint(pendingCheckpoint,
CheckpointFailureReason.JOB_FAILURE, cause);
Review comment:
Actually, `CheckpointFailureManager` can take more effect in the future but
not now. This is an intermediate step of the whole three PRs. In this PR, we
need to keep compatible with `setFailOnCheckpointingErrors`, it's the most
important thing. Otherwise, it will change many user behaviors. We have
considered counting more failure reason before, but it will make more changes
and make this PR more complex. So your thought is right but not for now.
The purpose of this PR is to introduce the `CheckpointFailureManager` and do
further residual refactor work for the first PR #7571.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] yanghua commented on a change in pull request #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure
yanghua commented on a change in pull request #8322: [FLINK-12364] Introduce a
CheckpointFailureManager to centralized manage checkpoint failure
URL: https://github.com/apache/flink/pull/8322#discussion_r294066885
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java
##
@@ -252,6 +255,22 @@ public void setFailOnCheckpointingErrors(boolean
failOnCheckpointingErrors) {
this.failOnCheckpointingErrors = failOnCheckpointingErrors;
}
+ /**
+* Get the tolerable checkpoint failure number which used by the
checkpoint failure manager
+* to determine when we need to fail the job.
+*/
+ public int getTolerableCheckpointFailureNumber() {
+ return tolerableCheckpointFailureNumber;
+ }
+
+ /**
+* Set the tolerable checkpoint failure number, the default value is 0
that means
+* we do not tolerance any checkpoint failure.
+*/
+ public void setTolerableCheckpointFailureNumber(int
tolerableCheckpointFailureNumber) {
Review comment:
IMO, you are wrong. `CheckpointingOptions` defines some global config
options which almost are related to the **state**. But
`tolerableCheckpointFailureNumber ` a checkpoint specific config option belongs
per job and it will enhance the config option of
`setFailOnCheckpointingErrors`. It should not be defined in
`CheckpointingOptions`. Currently, is it not used? Yes. Because we split this
feature into three steps. It will be used in the third step. But in this step,
we need it to create the `CheckpointFailureManager`. I think you know the
reason now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-12427) Translate the "Flink DataStream API Programming Guide" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864928#comment-16864928 ] xiaogang zhou commented on FLINK-12427: --- [~yangfei] any feedback laotie? > Translate the "Flink DataStream API Programming Guide" page into Chinese > > > Key: FLINK-12427 > URL: https://issues.apache.org/jira/browse/FLINK-12427 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.9.0 >Reporter: YangFei >Assignee: YangFei >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > > [https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html] > files locate /flink/docs/dev/datastream_api.zh.md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package
walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package URL: https://github.com/apache/flink/pull/8632#discussion_r294065932 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/params/shared/colname/HasOutputColName.java ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.params.shared.colname; + +import org.apache.flink.ml.api.misc.param.ParamInfo; +import org.apache.flink.ml.api.misc.param.ParamInfoFactory; +import org.apache.flink.ml.api.misc.param.WithParams; + +/** + * An interface for classes with a parameter specifying the name of the output column. Review comment: Reading over this comment I am not sure I understand what these interfaces are for. Also if we are ok to use scala. we might be able to group all of these traits in one file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package
walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package URL: https://github.com/apache/flink/pull/8632#discussion_r294065946 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/params/shared/colname/HasOutputColName.java ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.params.shared.colname; + +import org.apache.flink.ml.api.misc.param.ParamInfo; +import org.apache.flink.ml.api.misc.param.ParamInfoFactory; +import org.apache.flink.ml.api.misc.param.WithParams; + +/** + * An interface for classes with a parameter specifying the name of the output column. + * @see HasOutputColNames + * @see HasOutputColNameDvNull Review comment: Not sure I understand what `Dv` stands for. could we elaborate more? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package
walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package URL: https://github.com/apache/flink/pull/8632#discussion_r294065941 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/params/shared/colname/HasKeepColNames.java ## @@ -0,0 +1,44 @@ +/* Review comment: Can we have some example in `flink-ml-lib/src/test` ? this would be super helpful. Especially if one use case of "shared params" can be included. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package
walterddr commented on a change in pull request #8632: [FLINK-12744][ml] add shared params in ml package URL: https://github.com/apache/flink/pull/8632#discussion_r294065868 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/params/shared/colname/HasKeepColNames.java ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.params.shared.colname; + +import org.apache.flink.ml.api.misc.param.ParamInfo; +import org.apache.flink.ml.api.misc.param.ParamInfoFactory; +import org.apache.flink.ml.api.misc.param.WithParams; + +/** + * An interface for classes with a parameter specifying the names of the columns to be retained in the output table. Review comment: IMO `column` doesn't need to abbreviate. Also I prefer `fieldName` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8751: [FLINK-11937][StateBackend]Resolve small file problem in RocksDB incremental checkpoint
flinkbot commented on issue #8751: [FLINK-11937][StateBackend]Resolve small file problem in RocksDB incremental checkpoint URL: https://github.com/apache/flink/pull/8751#issuecomment-502415494 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] klion26 opened a new pull request #8751: [FLINK-11937][StateBackend]Resolve small file problem in RocksDB incremental checkpoint
klion26 opened a new pull request #8751: [FLINK-11937][StateBackend]Resolve small file problem in RocksDB incremental checkpoint URL: https://github.com/apache/flink/pull/8751 ## What is the purpose of the change This pr wants to fix the problem of too many small files in RocksDB incremental checkpoint. - commit 7aa56f5 introduces some components to support this feature - commit f2e8b43 modified some exist components to support this feature(one file for one checkpoint)and add some tests to verify this. ## Brief change log This pr introduces some necessary classes, such as - FsSegmentCheckpointStorage - FsCheckpointStreamFactory - FsSegmentCheckpointStorageLocation - FsSegmentCheckpointStreamFactory - FsSegmentStateBackend - FsSegmentStateHandle This pr modifies some exist classes to support this feature, such as: - SharedStateRegistry - RocksDBStateBackend and add some tests to verify the correct. ## Verifying this change This pr introduces some test and modified some exist test to verify the changes. - SharedStateRegistryTest - RocksDBStateBackendTest - FsSegmentCheckpointStorageTest - FsSegmentCheckpointStorageTest.java - FsSegmentStateBackendEntropyTest.java - FsSegmentStateHandleTest.java - RocksDBSegmentStateBackendFactoryTest.java - SavepointV2SerializerTest.java ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**no**) - The serializers: (**no**) - The runtime per-record code paths (performance sensitive): (**no**) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (**yes**) - The S3 file system connector: (**no**) ## Documentation - Does this pull request introduce a new feature? (**yes**) - If yes, how is the feature documented? (**JavaDocs**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11937) Resolve small file problem in RocksDB incremental checkpoint
[ https://issues.apache.org/jira/browse/FLINK-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11937: --- Labels: pull-request-available (was: ) > Resolve small file problem in RocksDB incremental checkpoint > > > Key: FLINK-11937 > URL: https://issues.apache.org/jira/browse/FLINK-11937 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing >Reporter: Congxian Qiu(klion26) >Assignee: Congxian Qiu(klion26) >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > > Currently when incremental checkpoint is enabled in RocksDBStateBackend a > separate file will be generated on DFS for each sst file. This may cause > “file flood” when running intensive workload (many jobs with high > parallelism) in big cluster. According to our observation in Alibaba > production, such file flood introduces at lease two drawbacks when using HDFS > as the checkpoint storage FileSystem: 1) huge number of RPC request issued to > NN which may burst its response queue; 2) huge number of files causes big > pressure on NN’s on-heap memory. > In Flink we ever noticed similar small file flood problem and tried to > resolved it by introducing ByteStreamStateHandle(FLINK-2808), but this > solution has its limitation that if we configure the threshold too low there > will still be too many small files, while if too high the JM will finally > OOM, thus could hardly resolve the issue in case of using RocksDBStateBackend > with incremental snapshot strategy. > We propose a new OutputStream called > FileSegmentCheckpointStateOutputStream(FSCSOS) to fix the problem. FSCSOS > will reuse the same underlying distributed file until its size exceeds a > preset threshold. We > plan to complete the work in 3 steps: firstly introduce FSCSOS, secondly > resolve the specific storage amplification issue on FSCSOS, and lastly add an > option to reuse FSCSOS across multiple checkpoints to further reduce the DFS > file number. > More details please refer to the attached design doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12859) ClassNotFoundException during MapRFs init
[ https://issues.apache.org/jira/browse/FLINK-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864926#comment-16864926 ] Aitozi commented on FLINK-12859: [~yunta] I repost the image. Do you have any idea about this ? > ClassNotFoundException during MapRFs init > - > > Key: FLINK-12859 > URL: https://issues.apache.org/jira/browse/FLINK-12859 > Project: Flink > Issue Type: Test > Components: FileSystems >Affects Versions: 1.8.0 >Reporter: Aitozi >Priority: Minor > Attachments: 20190616110004.jpg > > > I was running _mvn clean package_ locally on master branch to pass all the > test case, I ran into the _ClassNotFoundException_ as shown below, but I > check the [flink-mapr-fs][pom.xml], it has not been changed for a long time, > so is this always a failed test case? > !image-2019-06-15-23-56-56-158.png! > TestCase is: FileSystemAccessTest.testGetMapRFs() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12859) ClassNotFoundException during MapRFs init
[ https://issues.apache.org/jira/browse/FLINK-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aitozi updated FLINK-12859: --- Attachment: 20190616110004.jpg > ClassNotFoundException during MapRFs init > - > > Key: FLINK-12859 > URL: https://issues.apache.org/jira/browse/FLINK-12859 > Project: Flink > Issue Type: Test > Components: FileSystems >Affects Versions: 1.8.0 >Reporter: Aitozi >Priority: Minor > Attachments: 20190616110004.jpg > > > I was running _mvn clean package_ locally on master branch to pass all the > test case, I ran into the _ClassNotFoundException_ as shown below, but I > check the [flink-mapr-fs][pom.xml], it has not been changed for a long time, > so is this always a failed test case? > !image-2019-06-15-23-56-56-158.png! > TestCase is: FileSystemAccessTest.testGetMapRFs() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang reassigned FLINK-12848:
--
Assignee: (was: aloyszhang)
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang closed FLINK-12848.
--
Resolution: Fixed
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Assignee: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang updated FLINK-12848:
---
Fix Version/s: 1.9.0
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Assignee: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12860) Document openSSL certificate and algorithm restrictions
Nico Kruber created FLINK-12860: --- Summary: Document openSSL certificate and algorithm restrictions Key: FLINK-12860 URL: https://issues.apache.org/jira/browse/FLINK-12860 Project: Flink Issue Type: Sub-task Components: Documentation, Runtime / Network Affects Versions: 1.9.0 Reporter: Nico Kruber Assignee: Nico Kruber Netty's openSSL engine works slightly differently with respect to supported cipher algorithms and certificate storage formats. This should be documented before the 1.9 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] NicoK commented on issue #7368: [FLINK-10742][network] Let Netty use Flink's buffers directly in credit-based mode
NicoK commented on issue #7368: [FLINK-10742][network] Let Netty use Flink's buffers directly in credit-based mode URL: https://github.com/apache/flink/pull/7368#issuecomment-502408212 Thanks @gaoyunhaii for the in-depth analysis: these are very good results indeed - unfortunately, I won't have time next week to review this PR but will try to schedule for the week after if there is no other review until then. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-12517) Run network tests with dynamically-linked openSSL
[ https://issues.apache.org/jira/browse/FLINK-12517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber closed FLINK-12517. --- > Run network tests with dynamically-linked openSSL > - > > Key: FLINK-12517 > URL: https://issues.apache.org/jira/browse/FLINK-12517 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > FLINK-9816 adds the ability to work with Netty's wrapper around native > openSSL implementations. We should set up unit tests that verify the > artifacts we provide, i.e. the dynamically-linked openSSL one in flink-shaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12839) Package flink-shaded-netty-tcnative-dynamic into opt/
[ https://issues.apache.org/jira/browse/FLINK-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber closed FLINK-12839. --- > Package flink-shaded-netty-tcnative-dynamic into opt/ > - > > Key: FLINK-12839 > URL: https://issues.apache.org/jira/browse/FLINK-12839 > Project: Flink > Issue Type: Sub-task > Components: Build System, Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12518) Run e2e tests with openSSL
[ https://issues.apache.org/jira/browse/FLINK-12518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber closed FLINK-12518. --- > Run e2e tests with openSSL > -- > > Key: FLINK-12518 > URL: https://issues.apache.org/jira/browse/FLINK-12518 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network, Tests >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > We should modify one end-to-end test each to run with: > * Java-based SSL > * dynamically linked openSSL > * statically linked openSSL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12446) Add documentation how to enable openSSL
[ https://issues.apache.org/jira/browse/FLINK-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber closed FLINK-12446. --- > Add documentation how to enable openSSL > --- > > Key: FLINK-12446 > URL: https://issues.apache.org/jira/browse/FLINK-12446 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > Since we won't include openSSL support by default, we should describe the > (two) way of enabling this for Flink. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9816) Support Netty configuration to enable an openSSL-based SslEngine
[
https://issues.apache.org/jira/browse/FLINK-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber closed FLINK-9816.
--
> Support Netty configuration to enable an openSSL-based SslEngine
>
>
> Key: FLINK-9816
> URL: https://issues.apache.org/jira/browse/FLINK-9816
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Since a while now, Netty does not only support the JDK's {{SSLEngine}} but
> also implements one based on openSSL which, according to
> https://netty.io/wiki/requirements-for-4.x.html#wiki-h4-4 is significantly
> faster. We should add support for using that engine instead.
> This ticket is for adding the necessary parts to configure and setup an
> arbitrary Netty-supported SslEngine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-12838) Netty-fy the network SSL setup
[ https://issues.apache.org/jira/browse/FLINK-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber closed FLINK-12838. --- > Netty-fy the network SSL setup > -- > > Key: FLINK-12838 > URL: https://issues.apache.org/jira/browse/FLINK-12838 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > Refactor the SSL configuration done for Netty to have it more like the way > Netty intends it to be: using its SslContextBuilder. This will make it much > easier to set a different Netty SSL engine provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-11579) Support Netty SslEngine based on openSSL
[ https://issues.apache.org/jira/browse/FLINK-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-11579. - Resolution: Fixed Fix Version/s: 1.9.0 Release Note: You can now optionally use openSSL as Flink's SSL engine. According to https://netty.io/wiki/requirements-for-4.x.html#wiki-h4-4, this is significantly faster. Please refer to https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#security-ssl-provider for the available options and how to set openSSL up. > Support Netty SslEngine based on openSSL > > > Key: FLINK-11579 > URL: https://issues.apache.org/jira/browse/FLINK-11579 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > Since a while now, Netty does not only support the JDK's SSLEngine but also > implements one based on openSSL which, according to > https://netty.io/wiki/requirements-for-4.x.html#wiki-h4-4 is significantly > faster. We should add support for using that engine instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-12518) Run e2e tests with openSSL
[ https://issues.apache.org/jira/browse/FLINK-12518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-12518. - Resolution: Fixed Fix Version/s: 1.9.0 Fixed on master via - e5b4a57b0cdd6b01ac279c70b510c2f05e4e361b > Run e2e tests with openSSL > -- > > Key: FLINK-12518 > URL: https://issues.apache.org/jira/browse/FLINK-12518 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network, Tests >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > We should modify one end-to-end test each to run with: > * Java-based SSL > * dynamically linked openSSL > * statically linked openSSL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-12446) Add documentation how to enable openSSL
[ https://issues.apache.org/jira/browse/FLINK-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-12446. - Resolution: Fixed Fix Version/s: 1.9.0 Fixed on master via - 11a7b3f5802ed9aa0ad151ae3ff80244875cb4b5 > Add documentation how to enable openSSL > --- > > Key: FLINK-12446 > URL: https://issues.apache.org/jira/browse/FLINK-12446 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > Since we won't include openSSL support by default, we should describe the > (two) way of enabling this for Flink. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-12517) Run network tests with dynamically-linked openSSL
[ https://issues.apache.org/jira/browse/FLINK-12517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-12517. - Resolution: Fixed Fix Version/s: 1.9.0 Fixed on master via - d11be3358ac67f11d2859176412834d16a547301 > Run network tests with dynamically-linked openSSL > - > > Key: FLINK-12517 > URL: https://issues.apache.org/jira/browse/FLINK-12517 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > FLINK-9816 adds the ability to work with Netty's wrapper around native > openSSL implementations. We should set up unit tests that verify the > artifacts we provide, i.e. the dynamically-linked openSSL one in flink-shaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-9816) Support Netty configuration to enable an openSSL-based SslEngine
[
https://issues.apache.org/jira/browse/FLINK-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber resolved FLINK-9816.
Resolution: Fixed
Fix Version/s: 1.9.0
Fixed on master via
- ddfd4b89e06a4c89fa39f79ad2d3633c9dd09a85
> Support Netty configuration to enable an openSSL-based SslEngine
>
>
> Key: FLINK-9816
> URL: https://issues.apache.org/jira/browse/FLINK-9816
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Since a while now, Netty does not only support the JDK's {{SSLEngine}} but
> also implements one based on openSSL which, according to
> https://netty.io/wiki/requirements-for-4.x.html#wiki-h4-4 is significantly
> faster. We should add support for using that engine instead.
> This ticket is for adding the necessary parts to configure and setup an
> arbitrary Netty-supported SslEngine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (FLINK-12839) Package flink-shaded-netty-tcnative-dynamic into opt/
[ https://issues.apache.org/jira/browse/FLINK-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-12839. - Resolution: Fixed Fix Version/s: 1.9.0 Fixed on master via - c4ec9be6c8c1fca9633bb25a2a26f94338988445 > Package flink-shaded-netty-tcnative-dynamic into opt/ > - > > Key: FLINK-12839 > URL: https://issues.apache.org/jira/browse/FLINK-12839 > Project: Flink > Issue Type: Sub-task > Components: Build System, Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-12838) Netty-fy the network SSL setup
[ https://issues.apache.org/jira/browse/FLINK-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nico Kruber resolved FLINK-12838. - Resolution: Fixed Fix Version/s: 1.9.0 Fixed on master via - 65e822b67ec93e5cf50e6303c838c3b30935834a > Netty-fy the network SSL setup > -- > > Key: FLINK-12838 > URL: https://issues.apache.org/jira/browse/FLINK-12838 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Fix For: 1.9.0 > > > Refactor the SSL configuration done for Netty to have it more like the way > Netty intends it to be: using its SslContextBuilder. This will make it much > easier to set a different Netty SSL engine provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] NicoK merged pull request #7688: [FLINK-9816][network] add option to configure SSL engine provider for TM communication
NicoK merged pull request #7688: [FLINK-9816][network] add option to configure SSL engine provider for TM communication URL: https://github.com/apache/flink/pull/7688 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-4874) Document how to enable Flink web interface in local execution
[
https://issues.apache.org/jira/browse/FLINK-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864824#comment-16864824
]
Timothy Farkas commented on FLINK-4874:
---
[~fhueske] Can I pick up documenting this? I think it would be a good beginner
task for me.
> Document how to enable Flink web interface in local execution
> -
>
> Key: FLINK-4874
> URL: https://issues.apache.org/jira/browse/FLINK-4874
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.1.2, 1.2.0
>Reporter: Krishna Prasad Anna Ramesh Kumar
>Priority: Trivial
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> The local environment section in the local execution web page
> (https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/local_execution.html)
> indicates that in the web interface cannot be started while running in a
> local environment. As Till has pointed out in one of the mailing lists
> topics, this can be done by including the following in the program code.
> "Configuration config = new Configuration();
> config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironment(getP, config);"
> And adding this dependency
> "
> org.apache.flink
> flink-runtime-web_2.10
> ${flink.version}
> "
> This should be added to the documentation as it very critical for developers
> like time who are trying to learn the framework.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] Aitozi commented on issue #8455: [FLINK-12284, FLINK-12637][Network, Metrics]Fix the incorrect inputBufferUsage metric in credit-based network mode
Aitozi commented on issue #8455: [FLINK-12284,FLINK-12637][Network,Metrics]Fix the incorrect inputBufferUsage metric in credit-based network mode URL: https://github.com/apache/flink/pull/8455#issuecomment-502385780 Hi @pnowojski , I have adjust the test case according to your suggestion, please take a look when you have time, Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8750: [FLINK-11606] Translate the "Distributed Runtime Environment" page in to Chinese
flinkbot commented on issue #8750: [FLINK-11606] Translate the "Distributed Runtime Environment" page in to Chinese URL: https://github.com/apache/flink/pull/8750#issuecomment-502385761 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Flyangz opened a new pull request #8750: [FLINK-11606] Translate the "Distributed Runtime Environment" page in to Chinese
Flyangz opened a new pull request #8750: [FLINK-11606] Translate the "Distributed Runtime Environment" page in to Chinese URL: https://github.com/apache/flink/pull/8750 ## What is the purpose of the change This pull request completes the Chinese translation of "Distributed Runtime Environment" page from official document. ## Brief change log Translate the "Distributed Runtime Environment" page into Chinese ## Verifying this change This change is to add a new translated document. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12859) ClassNotFoundException during MapRFs init
[ https://issues.apache.org/jira/browse/FLINK-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864791#comment-16864791 ] Yun Tang commented on FLINK-12859: -- [~aitozi] I am afraid that the attached image has been broken to view. > ClassNotFoundException during MapRFs init > - > > Key: FLINK-12859 > URL: https://issues.apache.org/jira/browse/FLINK-12859 > Project: Flink > Issue Type: Test > Components: FileSystems >Affects Versions: 1.8.0 >Reporter: Aitozi >Priority: Minor > > I was running _mvn clean package_ locally on master branch to pass all the > test case, I ran into the _ClassNotFoundException_ as shown below, but I > check the [flink-mapr-fs][pom.xml], it has not been changed for a long time, > so is this always a failed test case? > !image-2019-06-15-23-56-56-158.png! > TestCase is: FileSystemAccessTest.testGetMapRFs() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] Myasuka commented on a change in pull request #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure
Myasuka commented on a change in pull request #8322: [FLINK-12364] Introduce a
CheckpointFailureManager to centralized manage checkpoint failure
URL: https://github.com/apache/flink/pull/8322#discussion_r294053711
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -1329,10 +1331,13 @@ private void discardCheckpoint(PendingCheckpoint
pendingCheckpoint, @Nullable Th
LOG.info("Discarding checkpoint {} of job {}.", checkpointId,
job, cause);
- if (cause == null || cause instanceof
CheckpointDeclineException) {
-
pendingCheckpoint.abort(CheckpointFailureReason.CHECKPOINT_DECLINED, cause);
+ if (cause == null) {
+ failPendingCheckpoint(pendingCheckpoint,
CheckpointFailureReason.CHECKPOINT_DECLINED);
+ } else if (cause instanceof CheckpointException) {
+ CheckpointException exception = (CheckpointException)
cause;
+ failPendingCheckpoint(pendingCheckpoint,
exception.getCheckpointFailureReason(), cause);
} else {
-
pendingCheckpoint.abort(CheckpointFailureReason.JOB_FAILURE, cause);
+ failPendingCheckpoint(pendingCheckpoint,
CheckpointFailureReason.JOB_FAILURE, cause);
Review comment:
I noticed that you actually did not touch the `AbstractStreamOperator` part.
With your PR, if operator fail to complete the snapshot in method
`#snapshotState(long , long , CheckpointOptions , CheckpointStreamFactory )`
and it would only decline that checkpoint and return as an `Exception`. In
other words, `CheckpointFailureManager` would only process the failed
checkpoint in this line which results in not increasing
`continuousFailureCounter` in `CheckpointFailureManager`. Is this reasonable?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] Myasuka commented on a change in pull request #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure
Myasuka commented on a change in pull request #8322: [FLINK-12364] Introduce a
CheckpointFailureManager to centralized manage checkpoint failure
URL: https://github.com/apache/flink/pull/8322#discussion_r294053508
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java
##
@@ -252,6 +255,22 @@ public void setFailOnCheckpointingErrors(boolean
failOnCheckpointingErrors) {
this.failOnCheckpointingErrors = failOnCheckpointingErrors;
}
+ /**
+* Get the tolerable checkpoint failure number which used by the
checkpoint failure manager
+* to determine when we need to fail the job.
+*/
+ public int getTolerableCheckpointFailureNumber() {
+ return tolerableCheckpointFailureNumber;
+ }
+
+ /**
+* Set the tolerable checkpoint failure number, the default value is 0
that means
+* we do not tolerance any checkpoint failure.
+*/
+ public void setTolerableCheckpointFailureNumber(int
tolerableCheckpointFailureNumber) {
Review comment:
I noticed that you have introduced variable
`tolerableCheckpointFailureNumber` and related setter here. However, I cannot
see any method to call this method. Why not introduce related configuration in
`CheckpointingOptions` so that user could configure this variable?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (FLINK-11606) Translate the "Distributed Runtime Environment" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-11606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859240#comment-16859240 ] Kevin Liu edited comment on FLINK-11606 at 6/15/19 4:52 PM: Hi [Huadong Sun|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Sunnyd], do you still have time to work on this task? If not, would you mind reappearing the issue? was (Author: lemony): Hi [Huadong Sun|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Sunnyd], do you still have time to work on this task? If not, would you mind reappearing the issue? Or [Jark Wu|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jark], do you have the right to do so? As I found that Huadong Sun hasn't had any new activities since he was assigned four months ago. And I'm very keen on this issue now. Thanks. !https://issues.apache.org/jira/images/icons/emoticons/smile.png! > Translate the "Distributed Runtime Environment" page into Chinese > - > > Key: FLINK-11606 > URL: https://issues.apache.org/jira/browse/FLINK-11606 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: Kevin Liu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/concepts/runtime.html > The markdown file is located in flink/docs/concepts/runtime.zh.md > The markdown file will be created once FLINK-11529 is merged. > You can reference the translation from : > https://github.com/flink-china/1.6.0/blob/master/concepts/runtime.md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10407) Reactive container mode
[ https://issues.apache.org/jira/browse/FLINK-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864753#comment-16864753 ] Flavio Junqueira commented on FLINK-10407: -- [~till.rohrmann] I was giving a look at the design document posted in this issue and I have a couple of questions. Is it ok if I post them here or is it better to do it on the sub-tasks or even the document itself? 1- According to the proposal, each operator can have its own rescaling policy, and it makes sense. I'm wondering about constraints across operators, though. Say that I give the target to an operator, but only the minimum to one or more operators downstream. Can it cause any runtime instability? If so, then perhaps we need also a way to specify constraints across operators. I'm interested in your feedback here. 2- It might be early to ask about the rescaling policy as there is little information about it in the document, but I'm interested in your current thoughts on how the interface is going to look like and how implementations are going to be used with a job. Specifically, I'm interested in how to connect the auto-scaling of [Pravega|http://github.com/pravega/pravega] to the Flink dynamic scaling so that we have scaling end-to-end. > Reactive container mode > --- > > Key: FLINK-10407 > URL: https://issues.apache.org/jira/browse/FLINK-10407 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > > The reactive container mode is a new operation mode where a Flink cluster > will react to newly available resources (e.g. started by an external service) > and make use of them by rescaling the existing job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] flinkbot commented on issue #8749: [hotfix][runtime] miscellaneous
flinkbot commented on issue #8749: [hotfix][runtime] miscellaneous URL: https://github.com/apache/flink/pull/8749#issuecomment-502378140 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-12859) ClassNotFoundException during MapRFs init
Aitozi created FLINK-12859: -- Summary: ClassNotFoundException during MapRFs init Key: FLINK-12859 URL: https://issues.apache.org/jira/browse/FLINK-12859 Project: Flink Issue Type: Test Components: FileSystems Affects Versions: 1.8.0 Reporter: Aitozi I was running _mvn clean package_ locally on master branch to pass all the test case, I ran into the _ClassNotFoundException_ as shown below, but I check the [flink-mapr-fs][pom.xml], it has not been changed for a long time, so is this always a failed test case? !image-2019-06-15-23-56-56-158.png! TestCase is: FileSystemAccessTest.testGetMapRFs() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] GJL opened a new pull request #8749: [hotfix][runtime] miscellaneous
GJL opened a new pull request #8749: [hotfix][runtime] miscellaneous URL: https://github.com/apache/flink/pull/8749 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #8707: [FLINK-12815] [table-planner-blink] Supports CatalogManager in blink planner
danny0405 commented on a change in pull request #8707: [FLINK-12815]
[table-planner-blink] Supports CatalogManager in blink planner
URL: https://github.com/apache/flink/pull/8707#discussion_r294047875
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/catalog/CatalogCalciteSchema.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.Internal;
+
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.calcite.schema.Function;
+import org.apache.calcite.schema.Schema;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.SchemaVersion;
+import org.apache.calcite.schema.Schemas;
+import org.apache.calcite.schema.Table;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * A mapping between Flink's catalog and Calcite's schema. This enables to
look up and access objects(tables, views,
+ * functions, types) in SQL queries without registering them in advance.
Databases are registered as sub-schemas
+ * in the schema.
+ */
+@Internal
+public class CatalogCalciteSchema implements Schema {
+
+ private final String catalogName;
+ private final Catalog catalog;
+
+ public CatalogCalciteSchema(String catalogName, Catalog catalog) {
+ this.catalogName = catalogName;
+ this.catalog = catalog;
+ }
+
+ /**
+* Look up a sub-schema (database) by the given sub-schema name.
+*
+* @param schemaName name of sub-schema to look up
+* @return the sub-schema with a given database name, or null
+*/
+ @Override
+ public Schema getSubSchema(String schemaName) {
+
Review comment:
This name `getSubSchema` seems a little misleading, since we always returns
a `DatabaseCalciteSchema`, i know that `getSubSchema` is a interface defined by
Calcite, but maybe we can have an abstraction of our own.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #8707: [FLINK-12815] [table-planner-blink] Supports CatalogManager in blink planner
danny0405 commented on a change in pull request #8707: [FLINK-12815] [table-planner-blink] Supports CatalogManager in blink planner URL: https://github.com/apache/flink/pull/8707#discussion_r294048274 ## File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/catalog/FlinkTempCatalogTable.java ## @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.catalog; + +import org.apache.flink.table.api.TableSchema; + +import org.apache.calcite.schema.impl.AbstractTable; + +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * Represents AbstractTable in Flink catalog. + */ Review comment: Please give more comments about why this special table is needed and how it should be used. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #8707: [FLINK-12815] [table-planner-blink] Supports CatalogManager in blink planner
danny0405 commented on a change in pull request #8707: [FLINK-12815]
[table-planner-blink] Supports CatalogManager in blink planner
URL: https://github.com/apache/flink/pull/8707#discussion_r294047994
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/catalog/CatalogManagerCalciteSchema.java
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.Internal;
+
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.calcite.schema.Function;
+import org.apache.calcite.schema.Schema;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.SchemaVersion;
+import org.apache.calcite.schema.Table;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Bridge between the {@link CatalogManager} and the {@link Schema}. This way
we can query Flink's specific catalogs
+ * from Calcite.
+ *
+ * The mapping for {@link Catalog}s is modeled as a strict two-level
reference structure for Flink in Calcite,
+ * the full path of objects is of format
[catalog_name].[db_name].[meta-object_name].
+ *
+ * It also supports {@link ExternalCatalog}s. An external catalog maps 1:1
to the Calcite's schema.
+ */
+@Internal
+public class CatalogManagerCalciteSchema implements Schema {
+
+ private final CatalogManager catalogManager;
+ private boolean isBatch;
+
Review comment:
Does this mean we can only have one kind of table for the whole
`CatalogManager` ? Just curious why the `CatalogManager` would take care of
this attribute, isn't this an attribute of just a single `table` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #8707: [FLINK-12815] [table-planner-blink] Supports CatalogManager in blink planner
danny0405 commented on a change in pull request #8707: [FLINK-12815]
[table-planner-blink] Supports CatalogManager in blink planner
URL: https://github.com/apache/flink/pull/8707#discussion_r294048193
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/catalog/DatabaseCalciteSchema.java
##
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.operations.DataStreamQueryOperation;
+import org.apache.flink.table.operations.QueryOperation;
+import org.apache.flink.table.plan.schema.TableSourceTable;
+import org.apache.flink.table.plan.stats.FlinkStatistic;
+
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.calcite.schema.Function;
+import org.apache.calcite.schema.Schema;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.SchemaVersion;
+import org.apache.calcite.schema.Schemas;
+import org.apache.calcite.schema.Table;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static java.lang.String.format;
+
+/**
+ * A mapping between Flink catalog's database and Calcite's schema.
+ * Tables are registered as tables in the schema.
+ */
+class DatabaseCalciteSchema implements Schema {
+ private final String databaseName;
+ private final String catalogName;
+ private final Catalog catalog;
+
+ public DatabaseCalciteSchema(String databaseName, String catalogName,
Catalog catalog) {
+ this.databaseName = databaseName;
+ this.catalogName = catalogName;
+ this.catalog = catalog;
+ }
+
+ @Override
+ public Table getTable(String tableName) {
+ ObjectPath tablePath = new ObjectPath(databaseName, tableName);
+
+ try {
+ if (!catalog.tableExists(tablePath)) {
+ return null;
+ }
+
+ CatalogBaseTable table = catalog.getTable(tablePath);
+
+ // TODO supports GenericCatalogView
+ if (table instanceof QueryOperationCatalogView) {
+ QueryOperationCatalogView view =
(QueryOperationCatalogView) table;
+ QueryOperation operation =
view.getQueryOperation();
+ if (operation instanceof
DataStreamQueryOperation) {
+ List qualifiedName =
Arrays.asList(catalogName, databaseName, tableName);
+ ((DataStreamQueryOperation)
operation).setQualifiedName(qualifiedName);
+ }
+ return
QueryOperationCatalogViewTable.createCalciteTable(view);
+ } else if (table instanceof ConnectorCatalogTable) {
+ ConnectorCatalogTable connectorTable =
(ConnectorCatalogTable) table;
+ return connectorTable.getTableSource()
+ .map(tableSource -> new
TableSourceTable<>(
+ tableSource,
+ !connectorTable.isBatch(),
+ FlinkStatistic.UNKNOWN())
+ ).orElseThrow(() -> new
TableException("Cannot query a sink only table."));
+ } else if (table instanceof FlinkTempCatalogTable) {
+ return ((FlinkTempCatalogTable)
table).getAbstractTable();
+ } else {
+ throw new TableException("Unsupported table
type: " + table);
+ }
+
[jira] [Created] (FLINK-12858) Potentially not properly working Flink job in case of stop-with-savepoint failure
Alex created FLINK-12858:
Summary: Potentially not properly working Flink job in case of
stop-with-savepoint failure
Key: FLINK-12858
URL: https://issues.apache.org/jira/browse/FLINK-12858
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Reporter: Alex
Current implementation of stop-with-savepoint (FLINK-11458) would lock the
thread (on {{syncSavepointLatch}}) that carries
{{StreamTask.performCheckpoint()}}. For non-source tasks, this thread is
implied to be the task's main thread (stop-with-savepoint deliberately stops
any activity in the task's main thread).
Unlocking happens either when the task is cancelled or when the corresponding
checkpoint is acknowledged.
It's possible, that other downstream tasks of the same Flink job "soft" fail
the checkpoint/savepoint due to various reasons (for example, due to max
buffered bytes {{BarrierBuffer.checkSizeLimit()}}. In such case, the checkpoint
abortion would be notified to JM . But it looks like, the checkpoint
coordinator would handle such abortion as usual and assume that the Flink job
continues running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-12143) Mechanism to ship plugin jars in the cluster
[
https://issues.apache.org/jira/browse/FLINK-12143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864085#comment-16864085
]
Alex edited comment on FLINK-12143 at 6/15/19 11:01 AM:
-The problem is because
* {{parentFirstLoaderPatterns}} contains {{org.apache.flink.}} as matching
prefix;
* the hadoop file system component loads some classes dynamically
({{java.lang.Class.forName}}) and one of them
{{org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.PropertiesConfiguration}}
matches the pattern.
This triggers to use the application global class loader, instead of searching
in the jar.
The test failure can be reproduced by slightly modifying {{PluginLoaderTest}}:
setting the matching {{loaderExcludePatterns}} on [line
42|[https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/plugin/PluginLoaderTest.java#L42]].
For example to:
{code:java}
// Use explicit prefix of org.apache.flink.test.plugin.jar.plugina.DynamicClassA
PluginDescriptor pluginDescriptorA = new PluginDescriptor("A", new
URL[]{classpathA}, new String[]{"org.apache.flink."});
{code}
Although, running this test in IDE would not throw {{ClassNotFound}} exception,
but the actual class lookup would use the global class loader.-
Update: as was pointed out by [~StephanEwen] and [~pnowojski], the root case is
due to usage of different class loader (retrieved by
{{Thread.currentThread().getContextClassLoader()}}) later in the execution path
(after a plugin has been loaded).
The agreed workaround is to additionally set plugin's class loader in the
thread context during file system creation.
was (Author: 1u0):
-The problem is because
* {{parentFirstLoaderPatterns}} contains {{org.apache.flink.}} as matching
prefix;
* the hadoop file system component loads some classes dynamically
({{java.lang.Class.forName}}) and one of them
{{org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.PropertiesConfiguration}}
matches the pattern.
This triggers to use the application global class loader, instead of searching
in the jar.
The test failure can be reproduced by slightly modifying {{PluginLoaderTest}}:
setting the matching {{loaderExcludePatterns}} on [line
42|[https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/plugin/PluginLoaderTest.java#L42]].
For example to:
{code:java}
// Use explicit prefix of org.apache.flink.test.plugin.jar.plugina.DynamicClassA
PluginDescriptor pluginDescriptorA = new PluginDescriptor("A", new
URL[]{classpathA}, new String[]{"org.apache.flink."});
{code}
Although, running this test in IDE would not throw {{ClassNotFound}} exception,
but the actual class lookup would use the global class loader.-
As was pointed out by [~StephanEwen] and [~pnowojski], the root case is due to
usage of different class loader (retrieved by
{{Thread.currentThread().getContextClassLoader()}}) later in the execution path
(after a plugin has been loaded).
The agreed workaround is to additionally set plugin's class loader in the
thread context during file system creation.
> Mechanism to ship plugin jars in the cluster
>
>
> Key: FLINK-12143
> URL: https://issues.apache.org/jira/browse/FLINK-12143
> Project: Flink
> Issue Type: Sub-task
> Components: FileSystems, Runtime / Coordination
>Reporter: Stefan Richter
>Assignee: Alex
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-12143) Mechanism to ship plugin jars in the cluster
[
https://issues.apache.org/jira/browse/FLINK-12143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864085#comment-16864085
]
Alex edited comment on FLINK-12143 at 6/15/19 11:00 AM:
-The problem is because
* {{parentFirstLoaderPatterns}} contains {{org.apache.flink.}} as matching
prefix;
* the hadoop file system component loads some classes dynamically
({{java.lang.Class.forName}}) and one of them
{{org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.PropertiesConfiguration}}
matches the pattern.
This triggers to use the application global class loader, instead of searching
in the jar.
The test failure can be reproduced by slightly modifying {{PluginLoaderTest}}:
setting the matching {{loaderExcludePatterns}} on [line
42|[https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/plugin/PluginLoaderTest.java#L42]].
For example to:
{code:java}
// Use explicit prefix of org.apache.flink.test.plugin.jar.plugina.DynamicClassA
PluginDescriptor pluginDescriptorA = new PluginDescriptor("A", new
URL[]{classpathA}, new String[]{"org.apache.flink."});
{code}
Although, running this test in IDE would not throw {{ClassNotFound}} exception,
but the actual class lookup would use the global class loader.-
As was pointed out by [~StephanEwen] and [~pnowojski], the root case is due to
usage of different class loader (retrieved by
{{Thread.currentThread().getContextClassLoader()}}) later in the execution path
(after a plugin has been loaded).
The agreed workaround is to additionally set plugin's class loader in the
thread context during file system creation.
was (Author: 1u0):
The problem is because
* {{parentFirstLoaderPatterns}} contains {{org.apache.flink.}} as matching
prefix;
* the hadoop file system component loads some classes dynamically
({{java.lang.Class.forName}}) and one of them
{{org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.PropertiesConfiguration}}
matches the pattern.
This triggers to use the application global class loader, instead of searching
in the jar.
The test failure can be reproduced by slightly modifying {{PluginLoaderTest}}:
setting the matching {{loaderExcludePatterns}} on [line
42|[https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/plugin/PluginLoaderTest.java#L42]].
For example to:
{code:java}
// Use explicit prefix of org.apache.flink.test.plugin.jar.plugina.DynamicClassA
PluginDescriptor pluginDescriptorA = new PluginDescriptor("A", new
URL[]{classpathA}, new String[]{"org.apache.flink."});
{code}
Although, running this test in IDE would not throw {{ClassNotFound}} exception,
but the actual class lookup would use the global class loader.
> Mechanism to ship plugin jars in the cluster
>
>
> Key: FLINK-12143
> URL: https://issues.apache.org/jira/browse/FLINK-12143
> Project: Flink
> Issue Type: Sub-task
> Components: FileSystems, Runtime / Coordination
>Reporter: Stefan Richter
>Assignee: Alex
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-12653) Keyed state backend fails to restore during rescaling
[
https://issues.apache.org/jira/browse/FLINK-12653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels reassigned FLINK-12653:
--
Assignee: (was: Maximilian Michels)
> Keyed state backend fails to restore during rescaling
> -
>
> Key: FLINK-12653
> URL: https://issues.apache.org/jira/browse/FLINK-12653
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Environment: Beam 2.12.0 or any other Beam version
> Flink >= 1.6
> Heap/Filesystem state backend (RocksDB works fine)
>Reporter: Maximilian Michels
>Priority: Critical
>
> The Flink Runner includes a test which verifies checkpoints/savepoints work
> correctly with Beam on Flink. When adding additional tests for
> scaleup/scaledown [1], I came across a bug with restoring the keyed state
> backend. After a fair amount of debugging Beam code and checking any
> potential issues with serializers, I think this could be a Flink issue.
> Steps to reproduce:
> 1. {{git clone https://github.com/mxm/beam}}
> 2. {{cd beam && git checkout savepoint-problem}}
> 3. {{./gradlew :runners:flink:1.6:test --tests
> "**.FlinkSavepointTest.testSavepointRestoreLegacy"}}
> Error:
> {noformat}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:192)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for DoFnOperator_76375152c4a81d5df72cf49e32c4ecb9_(4/4) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:137)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:279)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:133)
> ... 5 more
> Caused by: java.lang.RuntimeException: Invalid namespace string: ''
> at
> org.apache.beam.runners.core.StateNamespaces.fromString(StateNamespaces.java:245)
> at
> org.apache.beam.runners.core.TimerInternals$TimerDataCoder.decode(TimerInternals.java:246)
> at
> org.apache.beam.runners.core.TimerInternals$TimerDataCoder.decode(TimerInternals.java:221)
> at
> org.apache.beam.runners.flink.translation.types.CoderTypeSerializer.deserialize(CoderTypeSerializer.java:92)
> at
> org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:169)
> at
> org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:45)
> at
> org.apache.flink.runtime.state.KeyGroupPartitioner$PartitioningResultKeyGroupReader.readMappingsInKeyGroup(KeyGroupPartitioner.java:297)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.readKeyGroupStateData(HeapKeyedStateBackend.java:513)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.readStateHandleStateData(HeapKeyedStateBackend.java:474)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restorePartitionedState(HeapKeyedStateBackend.java:431)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restore(HeapKeyedStateBackend.java:370)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restore(HeapKeyedStateBackend.java:105)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:151)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:123)
> ... 7 more
> {noformat}
> It is possible to change the {{maxParallelism}} to other values. The
> following lead to failure:
> {noformat}
>options.setMaxParallelism(128); // default value
>options.setMaxParallelism(64);
> options.setMaxParallelism(118);
> {noformat}
> The following work fine:
> {noformat}
> options.setMaxParallelism(110);
> options.setMaxParallelism(63);
> options.setMaxParallelism(24);
> {noformat}
> [1]
> https://github.com/a
[jira] [Commented] (FLINK-12653) Keyed state backend fails to restore during rescaling
[
https://issues.apache.org/jira/browse/FLINK-12653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864652#comment-16864652
]
Maximilian Michels commented on FLINK-12653:
We have worked around this issue in Beam. I still think it makes sense to fix
this in Flink. For now I won't have time to submit a patch, so I'm unassigning
myself in case somebody else wants to fix this in the meantime.
> Keyed state backend fails to restore during rescaling
> -
>
> Key: FLINK-12653
> URL: https://issues.apache.org/jira/browse/FLINK-12653
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Environment: Beam 2.12.0 or any other Beam version
> Flink >= 1.6
> Heap/Filesystem state backend (RocksDB works fine)
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Critical
>
> The Flink Runner includes a test which verifies checkpoints/savepoints work
> correctly with Beam on Flink. When adding additional tests for
> scaleup/scaledown [1], I came across a bug with restoring the keyed state
> backend. After a fair amount of debugging Beam code and checking any
> potential issues with serializers, I think this could be a Flink issue.
> Steps to reproduce:
> 1. {{git clone https://github.com/mxm/beam}}
> 2. {{cd beam && git checkout savepoint-problem}}
> 3. {{./gradlew :runners:flink:1.6:test --tests
> "**.FlinkSavepointTest.testSavepointRestoreLegacy"}}
> Error:
> {noformat}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:192)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for DoFnOperator_76375152c4a81d5df72cf49e32c4ecb9_(4/4) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:137)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:279)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:133)
> ... 5 more
> Caused by: java.lang.RuntimeException: Invalid namespace string: ''
> at
> org.apache.beam.runners.core.StateNamespaces.fromString(StateNamespaces.java:245)
> at
> org.apache.beam.runners.core.TimerInternals$TimerDataCoder.decode(TimerInternals.java:246)
> at
> org.apache.beam.runners.core.TimerInternals$TimerDataCoder.decode(TimerInternals.java:221)
> at
> org.apache.beam.runners.flink.translation.types.CoderTypeSerializer.deserialize(CoderTypeSerializer.java:92)
> at
> org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:169)
> at
> org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:45)
> at
> org.apache.flink.runtime.state.KeyGroupPartitioner$PartitioningResultKeyGroupReader.readMappingsInKeyGroup(KeyGroupPartitioner.java:297)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.readKeyGroupStateData(HeapKeyedStateBackend.java:513)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.readStateHandleStateData(HeapKeyedStateBackend.java:474)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restorePartitionedState(HeapKeyedStateBackend.java:431)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restore(HeapKeyedStateBackend.java:370)
> at
> org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.restore(HeapKeyedStateBackend.java:105)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:151)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:123)
> ... 7 more
> {noformat}
> It is possible to change the {{maxParallelism}} to other values. The
> following lead to failure:
> {noformat}
>options.setMaxParallelism(128); // default value
>opti
[GitHub] [flink] flinkbot commented on issue #8748: [FLINK-12857] [table] move FilterableTableSource into flink-table-common
flinkbot commented on issue #8748: [FLINK-12857] [table] move FilterableTableSource into flink-table-common URL: https://github.com/apache/flink/pull/8748#issuecomment-502349549 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12857) move FilterableTableSource into flink-table-common
[ https://issues.apache.org/jira/browse/FLINK-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-12857: --- Labels: pull-request-available (was: ) > move FilterableTableSource into flink-table-common > -- > > Key: FLINK-12857 > URL: https://issues.apache.org/jira/browse/FLINK-12857 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > > move FilterableTableSource into flink-table-common, so that flink-planner and > blink-planner could use this interface both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] godfreyhe opened a new pull request #8748: [FLINK-12857] [table] move FilterableTableSource into flink-table-common
godfreyhe opened a new pull request #8748: [FLINK-12857] [table] move FilterableTableSource into flink-table-common URL: https://github.com/apache/flink/pull/8748 ## What is the purpose of the change *move FilterableTableSource into flink-table-common* ## Brief change log - *move FilterableTableSource into flink-table-common and convert it as Java* ## Verifying this change All existing tests should be passed ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12857) move FilterableTableSource into flink-table-common
[ https://issues.apache.org/jira/browse/FLINK-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he updated FLINK-12857: --- Description: move FilterableTableSource into flink-table-common, so that flink-planner and blink-planner could use this interface both. (was: move FilterableTableSource into flink-table-common, and flink-planner and blink-planner could use this interface both.) > move FilterableTableSource into flink-table-common > -- > > Key: FLINK-12857 > URL: https://issues.apache.org/jira/browse/FLINK-12857 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > > move FilterableTableSource into flink-table-common, so that flink-planner and > blink-planner could use this interface both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12857) move FilterableTableSource into flink-table-common
godfrey he created FLINK-12857: -- Summary: move FilterableTableSource into flink-table-common Key: FLINK-12857 URL: https://issues.apache.org/jira/browse/FLINK-12857 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: godfrey he Assignee: godfrey he move FilterableTableSource into flink-table-common, and flink-planner and blink-planner could use this interface both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12856) Introduce planner rule to push projection into TableSource
[ https://issues.apache.org/jira/browse/FLINK-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-12856: --- Labels: pull-request-available (was: ) > Introduce planner rule to push projection into TableSource > -- > > Key: FLINK-12856 > URL: https://issues.apache.org/jira/browse/FLINK-12856 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > > This issue aims to support push projection into ProjectableTableSource or > NestedFieldsProjectableTableSource to reduce output fields of a TableSource -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] flinkbot commented on issue #8747: [FLINK-12856] [table-planner-blink] Introduce planner rule to push projection into TableSource
flinkbot commented on issue #8747: [FLINK-12856] [table-planner-blink] Introduce planner rule to push projection into TableSource URL: https://github.com/apache/flink/pull/8747#issuecomment-502347754 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe opened a new pull request #8747: [FLINK-12856] [table-planner-blink] Introduce planner rule to push projection into TableSource
godfreyhe opened a new pull request #8747: [FLINK-12856] [table-planner-blink] Introduce planner rule to push projection into TableSource URL: https://github.com/apache/flink/pull/8747 ## What is the purpose of the change *[FLINK-12856] [table-planner-blink] Introduce planner rule to push projection into TableSource* ## Brief change log - *Added PushProjectIntoTableSourceScanRule that pushes projection into TableSource* - *Added RexNodeExtractor to extract used field from expressions* - *Added RexNodeRewriter to rewrite expressions based on new indexes* ## Verifying this change This change added tests and can be verified as follows: - *Added PushProjectIntoTableSourceScanRuleTest that validates logical plan after PushProjectIntoTableSourceScanRule is applied* - *Added TableSourceTest that validates the physical plan for the projectable table source* - *Added TableSourceITCase that validates the execution result for the projectable table source* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12856) Introduce planner rule to push projection into TableSource
[ https://issues.apache.org/jira/browse/FLINK-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he updated FLINK-12856: --- Description: This issue aims to support push projection into ProjectableTableSource or NestedFieldsProjectableTableSource to reduce output fields of a TableSource (was: This issue aims to support push projection into ProjectableTableSource to reduce output fields of a TableSource) > Introduce planner rule to push projection into TableSource > -- > > Key: FLINK-12856 > URL: https://issues.apache.org/jira/browse/FLINK-12856 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > > This issue aims to support push projection into ProjectableTableSource or > NestedFieldsProjectableTableSource to reduce output fields of a TableSource -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12856) Introduce planner rule to push projection into TableSource
[ https://issues.apache.org/jira/browse/FLINK-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he updated FLINK-12856: --- Summary: Introduce planner rule to push projection into TableSource (was: Introduce planner rule to push projection into ProjectableTableSource) > Introduce planner rule to push projection into TableSource > -- > > Key: FLINK-12856 > URL: https://issues.apache.org/jira/browse/FLINK-12856 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > > This issue aims to support push projection into ProjectableTableSource to > reduce output fields of a TableSource -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12856) Introduce planner rule to push projection into ProjectableTableSource
godfrey he created FLINK-12856: -- Summary: Introduce planner rule to push projection into ProjectableTableSource Key: FLINK-12856 URL: https://issues.apache.org/jira/browse/FLINK-12856 Project: Flink Issue Type: New Feature Components: Table SQL / Planner Reporter: godfrey he Assignee: godfrey he This issue aims support push projection into ProjectableTableSource to reduce output fields of a TableSource -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12856) Introduce planner rule to push projection into ProjectableTableSource
[ https://issues.apache.org/jira/browse/FLINK-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he updated FLINK-12856: --- Description: This issue aims to support push projection into ProjectableTableSource to reduce output fields of a TableSource (was: This issue aims support push projection into ProjectableTableSource to reduce output fields of a TableSource) > Introduce planner rule to push projection into ProjectableTableSource > - > > Key: FLINK-12856 > URL: https://issues.apache.org/jira/browse/FLINK-12856 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > > This issue aims to support push projection into ProjectableTableSource to > reduce output fields of a TableSource -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] flinkbot commented on issue #8746: [hotfix][FLINK-11120][table]fix the bug of timestampadd handles time
flinkbot commented on issue #8746: [hotfix][FLINK-11120][table]fix the bug of timestampadd handles time URL: https://github.com/apache/flink/pull/8746#issuecomment-502345997 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11120) The bug of timestampadd handles time
[
https://issues.apache.org/jira/browse/FLINK-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-11120:
---
Labels: pull-request-available (was: )
> The bug of timestampadd handles time
> -
>
> Key: FLINK-11120
> URL: https://issues.apache.org/jira/browse/FLINK-11120
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Forward Xu
>Assignee: Forward Xu
>Priority: Major
> Labels: pull-request-available
>
> The error occur when {{timestampadd(MINUTE, 1, time '01:00:00')}} is executed:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Long
> at org.apache.calcite.rex.RexBuilder.clean(RexBuilder.java:1520)
> at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1318)
> at
> org.apache.flink.table.codegen.ExpressionReducer.reduce(ExpressionReducer.scala:135)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:620)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:540)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:288)
> I think it should meet the following conditions:
> ||expression||Expect the result||
> |timestampadd(MINUTE, -1, time '00:00:00')|23:59:00|
> |timestampadd(MINUTE, 1, time '00:00:00')|00:01:00|
> |timestampadd(MINUTE, 1, time '23:59:59')|00:00:59|
> |timestampadd(SECOND, 1, time '23:59:59')|00:00:00|
> |timestampadd(HOUR, 1, time '23:59:59')|00:59:59|
> This problem seems to be a bug in calcite. I have submitted isuse to calcite.
> The following is the link.
> CALCITE-2699
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] XuQianJin-Stars opened a new pull request #8746: [hotfix][FLINK-11120][table]fix the bug of timestampadd handles time
XuQianJin-Stars opened a new pull request #8746:
[hotfix][FLINK-11120][table]fix the bug of timestampadd handles time
URL: https://github.com/apache/flink/pull/8746
## What is the purpose of the change
*Fix the casting problem for function TIMESTAMPADD in Table*
## Brief change log
- *In judging `ScalarOperators` `generateTemporalPlusMinus` function to add
the following the `date` conversion `timestamp` of logic, are as follows:*
```scala
case (TIME_WITHOUT_TIME_ZONE, INTERVAL_DAY_TIME) =>
generateOperatorIfNotNull(ctx, new TimeType(), left, right) {
(l, r) => s"($l $op ((int) ($r))) % ${MILLIS_PER_DAY}"
}
case (TIME_WITHOUT_TIME_ZONE, INTERVAL_YEAR_MONTH) =>
generateOperatorIfNotNull(ctx, new TimeType(), left, right) {
(l, r) => s"$l"
}
```
## Verifying this change
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *`TIMESTAMPADD` with time; returns a time value.The interval is positive
for `ScalarFunctionsTest`.
- *`TIMESTAMPADD` with time; returns a time value .The interval is negative
for `ScalarFunctionsTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no /
don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs /
not documented)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
