[GitHub] [flink] SteNicholas commented on a change in pull request #13983: [FLINK-19989][python] Add collect operation in Python DataStream API
SteNicholas commented on a change in pull request #13983: URL: https://github.com/apache/flink/pull/13983#discussion_r531542160 ## File path: flink-python/pyflink/datastream/data_stream.py ## @@ -123,7 +124,7 @@ def set_max_parallelism(self, max_parallelism: int) -> 'DataStream': self._j_data_stream.setMaxParallelism(max_parallelism) return self -def get_type(self) -> TypeInformation: +def get_type(self) -> WrapperTypeInfo: Review comment: @dianfu Because type checker of PyFlink cause this change. I didn't want to change `TypeInformation` to `WrapperTypeInfo`. IMO, the basic type is `WrapperTypeInfo` not `TypeInformation `. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #13983: [FLINK-19989][python] Add collect operation in Python DataStream API
SteNicholas commented on a change in pull request #13983:
URL: https://github.com/apache/flink/pull/13983#discussion_r531982527
##
File path:
flink-python/src/main/java/org/apache/flink/api/common/python/PythonBridgeUtils.java
##
@@ -266,6 +278,92 @@ private static Object getPickledBytesFromJavaObject(Object
obj, LogicalType data
}
}
+ public static Object getPickledBytesFromJavaObject(Object obj,
TypeInformation dataType) throws IOException {
+ Pickler pickler = new Pickler();
+ initialize();
+ if (obj == null) {
+ return new byte[0];
+ } else {
+ if (dataType instanceof SqlTimeTypeInfo) {
+ SqlTimeTypeInfo sqlTimeTypeInfo =
SqlTimeTypeInfo.getInfoFor(dataType.getTypeClass());
+ if (sqlTimeTypeInfo == DATE) {
+ long time;
+ if (obj instanceof LocalDate) {
+ time = ((LocalDate)
(obj)).toEpochDay();
+ } else {
+ time = ((Date)
obj).toLocalDate().toEpochDay();
Review comment:
@dianfu , for `DateType` and `DATE`, the object could both be pickled
directly, but for `DateType` this has the type check to pickle the Date object.
IMO, this should be unified for `DateType` and `DATE`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 7c554a8e388693a7f243f211726bbb763dddcb60 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10278) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10280) * df3a80a44692c8b13d0026673b463a451d44608d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10283) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 7c554a8e388693a7f243f211726bbb763dddcb60 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10278) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10280) * df3a80a44692c8b13d0026673b463a451d44608d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20288) Correct documentation about savepoint self-contained
[ https://issues.apache.org/jira/browse/FLINK-20288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239908#comment-17239908 ] Congxian Qiu commented on FLINK-20288: -- [~yunta] thanks for creating this issue, maybe this is duplicated with FLINK-19381? > Correct documentation about savepoint self-contained > > > Key: FLINK-20288 > URL: https://issues.apache.org/jira/browse/FLINK-20288 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / Checkpointing >Affects Versions: 1.11.0 >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Savepoint self-contained has been supported while the documentation still > remain as not supported, we should fix that description. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
flinkbot edited a comment on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324 ## CI report: * b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147257697) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798) * 58784051e00030de475f361d314c1c0473ff2b8a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] klion26 commented on pull request #13488: [FLINK-19381][docs] Fix docs for savepoint relocation
klion26 commented on pull request #13488: URL: https://github.com/apache/flink/pull/13488#issuecomment-735047072 kindly ping @NicoK This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] klion26 commented on pull request #14175: [FLINK-20288][docs] Correct documentation about savepoint self-contained
klion26 commented on pull request #14175: URL: https://github.com/apache/flink/pull/14175#issuecomment-735047044 @Myasuka thanks for your contribution, maybe this is duplicated with FLINK-19381/[PR](https://github.com/apache/flink/pull/13488) ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-20384) Broken Link in deployment/ha/kubernetes_ha.zh.md
[
https://issues.apache.org/jira/browse/FLINK-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Congxian Qiu resolved FLINK-20384.
--
Fix Version/s: 1.12.0
Resolution: Fixed
merged into master d5b15652fc85fe4b0929e7faf274b46c04b7e924
1.12 2e8d9b9489a1ea33cecdcfc4d84912d5c68c1bf0
> Broken Link in deployment/ha/kubernetes_ha.zh.md
>
>
> Key: FLINK-20384
> URL: https://issues.apache.org/jira/browse/FLINK-20384
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Documentation
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When executing the script build_docs.sh, it will throw the following
> exception:
> {code:java}
> Liquid Exception: Could not find document
> 'deployment/resource-providers/standalone/kubernetes.md' in tag 'link'. Make
> sure the document exists and the path is correct. in
> deployment/ha/kubernetes_ha.zh.md Could not find document
> 'deployment/resource-providers/standalone/kubernetes.md' in tag 'link'.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20384) Broken Link in deployment/ha/kubernetes_ha.zh.md
[
https://issues.apache.org/jira/browse/FLINK-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Congxian Qiu reassigned FLINK-20384:
Assignee: Huang Xingbo
> Broken Link in deployment/ha/kubernetes_ha.zh.md
>
>
> Key: FLINK-20384
> URL: https://issues.apache.org/jira/browse/FLINK-20384
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Documentation
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
>
> When executing the script build_docs.sh, it will throw the following
> exception:
> {code:java}
> Liquid Exception: Could not find document
> 'deployment/resource-providers/standalone/kubernetes.md' in tag 'link'. Make
> sure the document exists and the path is correct. in
> deployment/ha/kubernetes_ha.zh.md Could not find document
> 'deployment/resource-providers/standalone/kubernetes.md' in tag 'link'.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] klion26 merged pull request #14242: [FLINK-20384][doc] Fix Wrong Links in Some Chinese Docs
klion26 merged pull request #14242: URL: https://github.com/apache/flink/pull/14242 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 7c554a8e388693a7f243f211726bbb763dddcb60 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10278) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10280) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] V1ncentzzZ commented on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
V1ncentzzZ commented on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-735046296 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
flinkbot edited a comment on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324 ## CI report: * b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147257697) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798) * 58784051e00030de475f361d314c1c0473ff2b8a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
HuangZhenQiu commented on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-735044796 @tillrohrmann @zentol @azagrebin Rebased master. Please review it when you have time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 7c554a8e388693a7f243f211726bbb763dddcb60 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 452c291a21493b2ae4e722de1879b5a94fca9ca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9935) * 7c554a8e388693a7f243f211726bbb763dddcb60 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] V1ncentzzZ commented on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
V1ncentzzZ commented on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-735038657 Hi @wuchong , I removed entry `Streaming Execution Plan` and add some tests. Hi @godfreyhe , Can i submit another issue for docs? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 452c291a21493b2ae4e722de1879b5a94fca9ca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9935) * 7c554a8e388693a7f243f211726bbb763dddcb60 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20410) SQLClientSchemaRegistryITCase.testWriting failed with "Subject 'user_behavior' not found.; error code: 40401"
[
https://issues.apache.org/jira/browse/FLINK-20410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239868#comment-17239868
]
Dian Fu commented on FLINK-20410:
-
cc [~dwysakowicz]
> SQLClientSchemaRegistryITCase.testWriting failed with "Subject
> 'user_behavior' not found.; error code: 40401"
> -
>
> Key: FLINK-20410
> URL: https://issues.apache.org/jira/browse/FLINK-20410
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10276=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 2020-11-28T01:14:08.6444305Z Nov 28 01:14:08 [ERROR]
> testWriting(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 74.818 s <<< ERROR!
> 2020-11-28T01:14:08.6445353Z Nov 28 01:14:08
> io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
> Subject 'user_behavior' not found.; error code: 40401
> 2020-11-28T01:14:08.6446071Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
> 2020-11-28T01:14:08.6446910Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
> 2020-11-28T01:14:08.6447522Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:769)
> 2020-11-28T01:14:08.6448352Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:760)
> 2020-11-28T01:14:08.6449091Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364)
> 2020-11-28T01:14:08.6449878Z Nov 28 01:14:08 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testWriting(SQLClientSchemaRegistryITCase.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20410) SQLClientSchemaRegistryITCase.testWriting failed with "Subject 'user_behavior' not found.; error code: 40401"
[
https://issues.apache.org/jira/browse/FLINK-20410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-20410:
Priority: Critical (was: Major)
> SQLClientSchemaRegistryITCase.testWriting failed with "Subject
> 'user_behavior' not found.; error code: 40401"
> -
>
> Key: FLINK-20410
> URL: https://issues.apache.org/jira/browse/FLINK-20410
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10276=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 2020-11-28T01:14:08.6444305Z Nov 28 01:14:08 [ERROR]
> testWriting(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 74.818 s <<< ERROR!
> 2020-11-28T01:14:08.6445353Z Nov 28 01:14:08
> io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
> Subject 'user_behavior' not found.; error code: 40401
> 2020-11-28T01:14:08.6446071Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
> 2020-11-28T01:14:08.6446910Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
> 2020-11-28T01:14:08.6447522Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:769)
> 2020-11-28T01:14:08.6448352Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:760)
> 2020-11-28T01:14:08.6449091Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364)
> 2020-11-28T01:14:08.6449878Z Nov 28 01:14:08 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testWriting(SQLClientSchemaRegistryITCase.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20410) SQLClientSchemaRegistryITCase.testWriting failed with "Subject 'user_behavior' not found.; error code: 40401"
[
https://issues.apache.org/jira/browse/FLINK-20410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-20410:
Affects Version/s: 1.13.0
> SQLClientSchemaRegistryITCase.testWriting failed with "Subject
> 'user_behavior' not found.; error code: 40401"
> -
>
> Key: FLINK-20410
> URL: https://issues.apache.org/jira/browse/FLINK-20410
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10276=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 2020-11-28T01:14:08.6444305Z Nov 28 01:14:08 [ERROR]
> testWriting(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 74.818 s <<< ERROR!
> 2020-11-28T01:14:08.6445353Z Nov 28 01:14:08
> io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
> Subject 'user_behavior' not found.; error code: 40401
> 2020-11-28T01:14:08.6446071Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
> 2020-11-28T01:14:08.6446910Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
> 2020-11-28T01:14:08.6447522Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:769)
> 2020-11-28T01:14:08.6448352Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:760)
> 2020-11-28T01:14:08.6449091Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364)
> 2020-11-28T01:14:08.6449878Z Nov 28 01:14:08 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testWriting(SQLClientSchemaRegistryITCase.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20410) SQLClientSchemaRegistryITCase.testWriting failed with "Subject 'user_behavior' not found.; error code: 40401"
[
https://issues.apache.org/jira/browse/FLINK-20410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-20410:
Labels: test-stability (was: )
> SQLClientSchemaRegistryITCase.testWriting failed with "Subject
> 'user_behavior' not found.; error code: 40401"
> -
>
> Key: FLINK-20410
> URL: https://issues.apache.org/jira/browse/FLINK-20410
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10276=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 2020-11-28T01:14:08.6444305Z Nov 28 01:14:08 [ERROR]
> testWriting(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 74.818 s <<< ERROR!
> 2020-11-28T01:14:08.6445353Z Nov 28 01:14:08
> io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
> Subject 'user_behavior' not found.; error code: 40401
> 2020-11-28T01:14:08.6446071Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
> 2020-11-28T01:14:08.6446910Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
> 2020-11-28T01:14:08.6447522Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:769)
> 2020-11-28T01:14:08.6448352Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:760)
> 2020-11-28T01:14:08.6449091Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364)
> 2020-11-28T01:14:08.6449878Z Nov 28 01:14:08 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testWriting(SQLClientSchemaRegistryITCase.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20410) SQLClientSchemaRegistryITCase.testWriting failed with "Subject 'user_behavior' not found.; error code: 40401"
Dian Fu created FLINK-20410:
---
Summary: SQLClientSchemaRegistryITCase.testWriting failed with
"Subject 'user_behavior' not found.; error code: 40401"
Key: FLINK-20410
URL: https://issues.apache.org/jira/browse/FLINK-20410
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.12.0
Reporter: Dian Fu
Fix For: 1.12.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10276=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
{code}
2020-11-28T01:14:08.6444305Z Nov 28 01:14:08 [ERROR]
testWriting(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
Time elapsed: 74.818 s <<< ERROR!
2020-11-28T01:14:08.6445353Z Nov 28 01:14:08
io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
Subject 'user_behavior' not found.; error code: 40401
2020-11-28T01:14:08.6446071Z Nov 28 01:14:08at
io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
2020-11-28T01:14:08.6446910Z Nov 28 01:14:08at
io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
2020-11-28T01:14:08.6447522Z Nov 28 01:14:08at
io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:769)
2020-11-28T01:14:08.6448352Z Nov 28 01:14:08at
io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:760)
2020-11-28T01:14:08.6449091Z Nov 28 01:14:08at
io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364)
2020-11-28T01:14:08.6449878Z Nov 28 01:14:08at
org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testWriting(SQLClientSchemaRegistryITCase.java:195)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] 1996fanrui commented on pull request #13885: [FLINK-19911] add read buffer for input stream
1996fanrui commented on pull request #13885: URL: https://github.com/apache/flink/pull/13885#issuecomment-735033850 > Hi @1996fanrui , > > that's an interesting discovery and investigation that you did there! > > I think the approach on the filesystem level is also much better than the previous way. Let's try to not change any public API (FileSystem) as this would slow down the progress. > > I'd probably focus on Hadoop file systems entirely (for now). What I'd propose is the following: > > * Use `HadoopFsFactory#configure` to extract the buffer size and pass it to the `ctor` of all filesystems created by the factory. > * Use that default buffer size in `HadoopFileSystem#open(Path)` to call `#open(Path, int)`. > * `HadoopFileSystem#open(Path, int)` should use the buffer size both in the call to Hadoop and to wrap it as you did as in the `BufferedFSInputStream`. I dug a bit into the Hadoop code and noticed that the cache is by default just 4kb. So even if we have cache on top of it with 64kb, we would still need to ask Hadoop several times. > > So, that means you are not adding any new methods, but just modify existing ones. Hi @Myasuka @AHeise , Based on the above design, there are some questions that I hope to be answered: - `buffer size` is passed to HadoopFileSystem through Constructor. It means that HadoopFileSystem needs to add a new Constructor: `HadoopFileSystem(FileSystem, bufferSize)`. HadoopFsFactory will call the new constructor. - The old constructor `HadoopFileSystem(FileSystem)` is still called by other FsFactory. So other FsFactory will not be able to improve performance. For example: OSSFileSystemFactory. Question: Do other FsFactory related to HadoopFileSystem need to improve performance? If needed, are there other better designs. (Can there be a way not to modify many FsFactory?) Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20409) Migrate test_kubernetes_pyflink_application.sh
Huang Xingbo created FLINK-20409: Summary: Migrate test_kubernetes_pyflink_application.sh Key: FLINK-20409 URL: https://issues.apache.org/jira/browse/FLINK-20409 Project: Flink Issue Type: Sub-task Components: API / Python, Tests Reporter: Huang Xingbo -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20408) Migrate test_pyflink_yarn.sh
Huang Xingbo created FLINK-20408: Summary: Migrate test_pyflink_yarn.sh Key: FLINK-20408 URL: https://issues.apache.org/jira/browse/FLINK-20408 Project: Flink Issue Type: Sub-task Components: API / Python, Tests Reporter: Huang Xingbo -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20407) Migrate test_pyflink.sh
Huang Xingbo created FLINK-20407: Summary: Migrate test_pyflink.sh Key: FLINK-20407 URL: https://issues.apache.org/jira/browse/FLINK-20407 Project: Flink Issue Type: Sub-task Components: API / Python, Tests Reporter: Huang Xingbo -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r531823381 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -30,83 +32,88 @@ Each file must include the Apache license information as a header. */ ``` -## 2. Tools + + -We recommend to follow the [IDE Setup Guide](https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/ide_setup.html#checkstyle-for-java) to get IDE tooling configured. +## 2. 工具 + +我们建议你按照 [IDE 设置指南](https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/ide_setup.html#checkstyle-for-java) 配置 IDE 工具。 -### Warnings +### 警告 + +* 我们争取实现零警告 +* 尽管现有的代码中存在许多警告，但新的修改不应该有任何编译器警告 +* 如果不能用合理的方式处理警告（某些情况下使用泛型时）也应该添加注释以压制警告 +* 弃用方法时，检查是否会引入其他的警告 -* We strive for zero warnings -* Even though there are many warnings in existing code, new changes should not add any additional compiler warnings -* If it is not possible to address the warning in a sane way (in some cases when working with generics) add an annotation to suppress the warning -* When deprecating methods, check that this does not introduce additional warnings + -## 3. Comments And Code Readability +## 3. 注释和代码可读性 -### Comments +### 注释 -**Golden rule: Comment as much as necessary to support code understanding, but don’t add redundant information.** +**黄金法则: 尽可能多的注释以支持代码的理解，但不要添加多余的信息。** -Think about +思考 -* What is the code doing? -* How does the code do this? -* Why is the code like that? +* What 代码在做什么？ +* How 代码怎么做到的？ +* Why 代码为什么是这样的？ -The code alone should explain as much as possible the “what” and the “how” +代码本身应该尽可能的解释 “what” 和 “how” -* Use JavaDocs to describe the roles of classes and the contracts of methods, in cases where the contract is not obvious or intuitive from the method name (the “what”). -* The flow of the code should give a good description of the “how”. -Think of variable and method names as part of the code documenting itself. -* It often makes reading the code easier if larger blocks that form a unit are moved into a private method with a descriptive name of what that block is doing +* 使用 JavaDocs 来描述类的作用和方法的协议，以防止不能从方法名看出协议（“what”）。 +* 代码流程应该能够很好的描述 “how”。 Review comment: ```suggestion * 代码流程应该能够很好的描述 “how”，能够将变量和方法名看作是代码文档的一部分。 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r531823305 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -30,95 +32,101 @@ Each file must include the Apache license information as a header. */ ``` -## 2. Tools + + + -We recommend to follow the [IDE Setup Guide](https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/ide_setup.html#checkstyle-for-java) to get IDE tooling configured. +## 2. 工具 + +我们建议你按照 [IDE 设置指南](https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/ide_setup.html#checkstyle-for-java) 配置 IDE 工具。 -### Warnings +### 警告 + +* 我们争取实现零警告 +* 尽管现有的代码中存在许多警告，但新的修改不应该有任何编译器警告 +* 如果不能用合理的方式处理警告（某些情况下使用泛型时）也应该添加注释以压制警告 +* 弃用方法时，检查是否会引入其他的警告 -* We strive for zero warnings -* Even though there are many warnings in existing code, new changes should not add any additional compiler warnings -* If it is not possible to address the warning in a sane way (in some cases when working with generics) add an annotation to suppress the warning -* When deprecating methods, check that this does not introduce additional warnings + -## 3. Comments And Code Readability +## 3. 注释和代码可读性 -### Comments +### 注释 -**Golden rule: Comment as much as necessary to support code understanding, but don’t add redundant information.** +**黄金法则: 尽可能多的注释以支持代码的理解，但不要添加多余的信息。** -Think about +思考 -* What is the code doing? -* How does the code do this? -* Why is the code like that? +* What 代码在做什么？ +* How 代码怎么做到的？ +* Why 代码为什么是这样的？ -The code alone should explain as much as possible the “what” and the “how” +代码本身应该尽可能的解释 “what” 和 “how” -* Use JavaDocs to describe the roles of classes and the contracts of methods, in cases where the contract is not obvious or intuitive from the method name (the “what”). -* The flow of the code should give a good description of the “how”. -Think of variable and method names as part of the code documenting itself. -* It often makes reading the code easier if larger blocks that form a unit are moved into a private method with a descriptive name of what that block is doing +* 使用 JavaDocs 来描述类的作用和方法的协议，以防止不能从方法名看出协议（“what”）。 +* 代码流程应该能够很好的描述 “how”。 +将变量和方法名看作是代码文档的一部分。 Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394:
URL: https://github.com/apache/flink-web/pull/394#discussion_r531822683
##
File path: contributing/code-style-and-quality-common.zh.md
##
@@ -30,95 +32,101 @@ Each file must include the Apache license information as a
header.
*/
```
-## 2. Tools
+
+
+
-We recommend to follow the [IDE Setup
Guide](https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/ide_setup.html#checkstyle-for-java)
to get IDE tooling configured.
+## 2. 工具
+
+我们建议你按照 [IDE
设置指南](https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/ide_setup.html#checkstyle-for-java)
配置 IDE 工具。
-### Warnings
+### 警告
+
+* 我们争取实现零警告
+* 尽管现有的代码中存在许多警告，但新的修改不应该有任何编译器警告
+* 如果不能用合理的方式处理警告（某些情况下使用泛型时）也应该添加注释以压制警告
+* 弃用方法时，检查是否会引入其他的警告
-* We strive for zero warnings
-* Even though there are many warnings in existing code, new changes should not
add any additional compiler warnings
-* If it is not possible to address the warning in a sane way (in some cases
when working with generics) add an annotation to suppress the warning
-* When deprecating methods, check that this does not introduce additional
warnings
+
-## 3. Comments And Code Readability
+## 3. 注释和代码可读性
-### Comments
+### 注释
-**Golden rule: Comment as much as necessary to support code understanding, but
don’t add redundant information.**
+**黄金法则: 尽可能多的注释以支持代码的理解，但不要添加多余的信息。**
-Think about
+思考
-* What is the code doing?
-* How does the code do this?
-* Why is the code like that?
+* What 代码在做什么？
+* How 代码怎么做到的？
+* Why 代码为什么是这样的？
-The code alone should explain as much as possible the “what” and the “how”
+代码本身应该尽可能的解释 “what” 和 “how”
-* Use JavaDocs to describe the roles of classes and the contracts of methods,
in cases where the contract is not obvious or intuitive from the method name
(the “what”).
-* The flow of the code should give a good description of the “how”.
-Think of variable and method names as part of the code documenting itself.
-* It often makes reading the code easier if larger blocks that form a unit are
moved into a private method with a descriptive name of what that block is doing
+* 使用 JavaDocs 来描述类的作用和方法的协议，以防止不能从方法名看出协议（“what”）。
+* 代码流程应该能够很好的描述 “how”。
+将变量和方法名看作是代码文档的一部分。
+* 如果将组成单元较大块的代码移动到 private 方法中，并且该方法具有描述性的名称，那么代码的可读性就会更强。
-In-code comments help explain the “why”
+代码内部的注释有助于解释 “why”
-* For example `// this specific code layout helps the JIT to better do this or
that`
-* Or `// nulling out this field here means future write attempts are fail-fast`
-* Or `// for arguments with which this method is actually called, this
seemingly naive approach works actually better than any optimized/smart version`
+* 例如 `// 这种特定的代码布局可以让 JIT 更好的进行工作`
+* 或 `// 此字段为空将会导致写入尝试 fail-fast`
+* 或 `// 用于实际调用该方法的参数，这种看似简单的方式实际上比任何优化/智能版本更好`
-In-code comments should not state redundant information about the “what” and
“how” that is already obvious in the code itself.
+在代码注释中，不应该有关于 “what” 和 “how” 这么明显的冗余信息。
-JavaDocs should not state meaningless information (just to satisfy the
Checkstyle checker).
+JavaDocs 不应该说明无意义的信息 （这么做只是为了满足 Checkstyle 的检查）。
-__Don’t:__
+__反例：__
```
/**
- * The symbol expression.
+ * 符号表达式。
*/
public class CommonSymbolExpression {}
```
-__Do:__
+__正例：__
```
/**
- * An expression that wraps a single specific symbol.
- * A symbol could be a unit, an alias, a variable, etc.
+ * 包含单个特定符号的表达式。
+ * 符号可以是 Unit、Alias、Variable 等等。
*/
public class CommonSymbolExpression {}
```
-### Branches and Nesting
+### 分支和嵌套
-Avoid deep nesting of scopes, by flipping the if condition and exiting early.
+通过对 if 条件取反并提前退出，避免超出嵌套深度的范围。
-__Don’t:__
+__反例：__
```
if (a) {
-if (b) {
+if (b) {
Review comment:
```suggestion
if (b) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r531822294 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -30,95 +32,101 @@ Each file must include the Apache license information as a header. */ ``` -## 2. Tools + + + Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] sjwiesman closed pull request #180: [FLINK-18810][sdk] Golang remote functions SDK
sjwiesman closed pull request #180: URL: https://github.com/apache/flink-statefun/pull/180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18810) Golang remote functions SDK
[ https://issues.apache.org/jira/browse/FLINK-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18810: --- Labels: pull-request-available (was: ) > Golang remote functions SDK > --- > > Key: FLINK-18810 > URL: https://issues.apache.org/jira/browse/FLINK-18810 > Project: Flink > Issue Type: New Feature > Components: Stateful Functions >Affects Versions: statefun-2.3.0 >Reporter: Francesco Guardiani >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Hi, > I was wondering if there's already some WIP for a Golang SDK to create remote > functions. If not, I'm willing to give it a try. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] sjwiesman opened a new pull request #180: [FLINK-18810][sdk] Golang remote functions SDK
sjwiesman opened a new pull request #180: URL: https://github.com/apache/flink-statefun/pull/180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14238: [FLINK-20347] Rework YARN documentation page
flinkbot edited a comment on pull request #14238: URL: https://github.com/apache/flink/pull/14238#issuecomment-734336545 ## CI report: * 5b9ca356af610ffa7faf50a88372a5e22f7c7185 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10273) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14252: [BP-1.12][FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14252: URL: https://github.com/apache/flink/pull/14252#issuecomment-734817896 ## CI report: * 9bff590490e977520cf92133855bd8a889f31fb4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10271) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14251: [FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14251: URL: https://github.com/apache/flink/pull/14251#issuecomment-734807691 ## CI report: * 69beddd215cef8366db8a9c9e71b0dad0469b22d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10270) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14253: [BP-1.11][FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14253: URL: https://github.com/apache/flink/pull/14253#issuecomment-734818578 ## CI report: * 6389f5ba95deedd53497b087df8da73d49d12929 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10272) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20406) Return the Checkpoint ID of the restored Checkpoint in CheckpointCoordinator.restoreLatestCheckpointedStateToSubtasks()
Stephan Ewen created FLINK-20406: Summary: Return the Checkpoint ID of the restored Checkpoint in CheckpointCoordinator.restoreLatestCheckpointedStateToSubtasks() Key: FLINK-20406 URL: https://issues.apache.org/jira/browse/FLINK-20406 Project: Flink Issue Type: Sub-task Components: Runtime / Checkpointing Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 1.12.0 To allow the scheduler to notify Operator Coordinators of subtask restores (local failover), we need to know which checkpoint ID was restored. This change does not adjust the other restore methods of the Checkpoint Coordinator, because the fact that the Scheduler needs to be involved in the subtask restore notification at all is only due to a shortcoming of the Checkpoint Coordinator: The CC is not aware of subtask restores, it always restores all subtasks and relies on the fact that assigning state to a running execution attempt has no effect. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * 6226047b9ce026f6431f9b8f7eed7ab49b373170 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10268) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20389) UnalignedCheckpointITCase failure caused by NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-20389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-20389:
---
Assignee: Matthias (was: Arvid Heise)
> UnalignedCheckpointITCase failure caused by NullPointerException
>
>
> Key: FLINK-20389
> URL: https://issues.apache.org/jira/browse/FLINK-20389
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Matthias
>Assignee: Matthias
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> [Build|https://dev.azure.com/mapohl/flink/_build/results?buildId=118=results]
> failed due to {{UnalignedCheckpointITCase}} caused by a
> {{NullPointerException}}:
> {code:java}
> Test execute[Parallel cogroup, p =
> 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) failed
> with:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:119)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:996)
> at akka.dispatch.OnComplete.internal(Future.scala:264)
> at akka.dispatch.OnComplete.internal(Future.scala:261)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=5,
> backoffTimeMS=100)
> at
>
[jira] [Commented] (FLINK-20389) UnalignedCheckpointITCase failure caused by NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-20389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239815#comment-17239815
]
Arvid Heise commented on FLINK-20389:
-
Improved UCITCase to not fail with NPE on unexpected exeception, for the actual
exception, [~mapohl] is investigating.
Merged improvement into master as f5889bdc0f63ae97859743dc73f5c675dfa7bd2f and
into 1.12 as 4744fb9de0.
> UnalignedCheckpointITCase failure caused by NullPointerException
>
>
> Key: FLINK-20389
> URL: https://issues.apache.org/jira/browse/FLINK-20389
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Matthias
>Assignee: Arvid Heise
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> [Build|https://dev.azure.com/mapohl/flink/_build/results?buildId=118=results]
> failed due to {{UnalignedCheckpointITCase}} caused by a
> {{NullPointerException}}:
> {code:java}
> Test execute[Parallel cogroup, p =
> 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) failed
> with:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:119)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:996)
> at akka.dispatch.OnComplete.internal(Future.scala:264)
> at akka.dispatch.OnComplete.internal(Future.scala:261)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
[GitHub] [flink] AHeise merged pull request #14250: [FLINK-20389][tests] Fix UnalignedCheckpointITCase to work with unassigned splits.
AHeise merged pull request #14250: URL: https://github.com/apache/flink/pull/14250 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
flinkbot edited a comment on pull request #14254: URL: https://github.com/apache/flink/pull/14254#issuecomment-734942786 ## CI report: * 9f8f123a323372be3edd985c83d8af11528dec16 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10269) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14238: [FLINK-20347] Rework YARN documentation page
flinkbot edited a comment on pull request #14238: URL: https://github.com/apache/flink/pull/14238#issuecomment-734336545 ## CI report: * 7806bb7e0dbca1263ddffb771dec367807f0590a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10261) * 5b9ca356af610ffa7faf50a88372a5e22f7c7185 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10273) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * b4d5fbae5b72a01ab9e94bc000373d6797339f9a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10266) * 6226047b9ce026f6431f9b8f7eed7ab49b373170 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10268) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
flinkbot edited a comment on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-732845197 ## CI report: * 6ffe3e161a8ec42b7ce5da3c46b57587d98a1f90 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10267) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14253: [BP-1.11][FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14253: URL: https://github.com/apache/flink/pull/14253#issuecomment-734818578 ## CI report: * f7bda2acaafa5f56911c79b02929157ab89b63aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10259) * 6389f5ba95deedd53497b087df8da73d49d12929 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10272) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14251: [FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14251: URL: https://github.com/apache/flink/pull/14251#issuecomment-734807691 ## CI report: * aa9069b35f492d5b4b982476db0205c1b0fd1fd7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10256) * 69beddd215cef8366db8a9c9e71b0dad0469b22d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10270) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14252: [BP-1.12][FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14252: URL: https://github.com/apache/flink/pull/14252#issuecomment-734817896 ## CI report: * f246a660e140fd3998c00363624205be5fb9801e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10258) * 9bff590490e977520cf92133855bd8a889f31fb4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10271) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14238: [FLINK-20347] Rework YARN documentation page
flinkbot edited a comment on pull request #14238: URL: https://github.com/apache/flink/pull/14238#issuecomment-734336545 ## CI report: * 7806bb7e0dbca1263ddffb771dec367807f0590a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10261) * 5b9ca356af610ffa7faf50a88372a5e22f7c7185 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
flinkbot edited a comment on pull request #14254: URL: https://github.com/apache/flink/pull/14254#issuecomment-734942786 ## CI report: * 9f8f123a323372be3edd985c83d8af11528dec16 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10269) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14253: [BP-1.11][FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14253: URL: https://github.com/apache/flink/pull/14253#issuecomment-734818578 ## CI report: * f7bda2acaafa5f56911c79b02929157ab89b63aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10259) * 6389f5ba95deedd53497b087df8da73d49d12929 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14252: [BP-1.12][FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14252: URL: https://github.com/apache/flink/pull/14252#issuecomment-734817896 ## CI report: * f246a660e140fd3998c00363624205be5fb9801e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10258) * 9bff590490e977520cf92133855bd8a889f31fb4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14251: [FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
flinkbot edited a comment on pull request #14251: URL: https://github.com/apache/flink/pull/14251#issuecomment-734807691 ## CI report: * aa9069b35f492d5b4b982476db0205c1b0fd1fd7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10256) * 69beddd215cef8366db8a9c9e71b0dad0469b22d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on pull request #14238: URL: https://github.com/apache/flink/pull/14238#issuecomment-734947167 Thanks a lot for the through review. I've addressed all comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531726061
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[jira] [Commented] (FLINK-20390) Programmatic access to the back-pressure
[ https://issues.apache.org/jira/browse/FLINK-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239806#comment-17239806 ] Steven Zhen Wu commented on FLINK-20390: not sure what is the input/source. If it is Kafka, would Kafka consumer lag (either in the number of msgs or in the wall clock time) be a good trigger. if it is lagging too much, it means that the job can't keep up with the input load. Then start to drop/sample msgs. > Programmatic access to the back-pressure > > > Key: FLINK-20390 > URL: https://issues.apache.org/jira/browse/FLINK-20390 > Project: Flink > Issue Type: New Feature > Components: API / Core >Reporter: Gaël Renoux >Priority: Major > > It would be useful to access the back-pressure monitoring from within > functions. > Here is our use case: we have a real-time Flink job, which takes decisions > based on input data. Sometimes, we have traffic spikes on the input and the > decisions process cannot processe records fast enough. Back-pressure starts > mounting, all the way back to the Source. What we want to do is to start > dropping records in this case, because it's better to make decisions based on > just a sample of the data rather than accumulate too much lag. > Right now, the only way is to have a filter with a hard-limit on the number > of records per-interval-of-time, and to drop records once we are over this > limit. However, this requires a lot of tuning to find out what the correct > limit is, especially since it may depend on the nature of the inputs (some > decisions take longer to make than others). It's also heavily dependent on > the buffers: the limit needs to be low enough that all records that pass the > limit can fit in the downstream buffers, or the back-pressure will will go > back past the filtering task and we're back to square one. Finally, it's not > very resilient to change: whenever we scale the infrastructure up, we need to > redo the whole tuning thing. > With programmatic access to the back-pressure, we could simply start dropping > records based on its current level. No tuning, and adjusted to the actual > issue. For performance, I assume it would be better if it reused the existing > back-pressure monitoring mechanism, rather than looking directly into the > buffer. A sampling of the back-pressure should be enough, and if more > precision is needed you can simply change the existing back-pressure > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20405) The LAG function in over window is not implemented correctly
[
https://issues.apache.org/jira/browse/FLINK-20405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-20405:
-
Summary: The LAG function in over window is not implemented correctly
(was: The LAG function in over window is not implements correctly)
> The LAG function in over window is not implemented correctly
>
>
> Key: FLINK-20405
> URL: https://issues.apache.org/jira/browse/FLINK-20405
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0
>Reporter: Leonard Xu
>Priority: Major
>
> For LAG(input, offset, default) function in over window, it always return
> current row's input no matter how the offset is set.
> After see the codegen code of the function, I think the implementation is not
> correct and need to correct.
> {code:java}
> // the offset and default value is never used
> public UnboundedOverAggregateHelper$24(java.lang.Object[] references) throws
> Exception {constant$14 = ((int) 1);
> constant$14isNull = false;
> constant$15 = ((org.apache.flink.table.data.binary.BinaryStringData)
> str$13);constant$15isNull = false;
> typeSerializer$19 =
> (((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
> references[0])); }
> public void accumulate(org.apache.flink.table.data.RowData accInput) throws
> Exception {
> org.apache.flink.table.data.binary.BinaryStringData field$21;
> boolean isNull$21;
> org.apache.flink.table.data.binary.BinaryStringData field$22;
> isNull$21 = accInput.isNullAt(2);field$21 =
> org.apache.flink.table.data.binary.BinaryStringData.EMPTY_UTF8;if
> (!isNull$21) { field$21 =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> accInput.getString(2));}field$22 = field$21;
> if (!isNull$21) { field$22 =
> (org.apache.flink.table.data.binary.BinaryStringData)
> (typeSerializer$19.copy(field$22));}
> if (agg0_leadlag != field$22) { agg0_leadlag =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> typeSerializer$19.copy(field$22));} ;
> agg0_leadlagIsNull = isNull$21; }
> {code}
>
> The question comes from user mail list[1]
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkSQL-kafka-gt-dedup-gt-kafka-td39335.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
flinkbot commented on pull request #14254: URL: https://github.com/apache/flink/pull/14254#issuecomment-734942786 ## CI report: * 9f8f123a323372be3edd985c83d8af11528dec16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14249: [FLINK-20391] Set FORWARD_EDGES_PIPELINED for BATCH ExecutionMode
flinkbot edited a comment on pull request #14249: URL: https://github.com/apache/flink/pull/14249#issuecomment-734789081 ## CI report: * 2ce421de97821f5fa47a965362d3c08c4971fa92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10260) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14251: [FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
tillrohrmann commented on a change in pull request #14251:
URL: https://github.com/apache/flink/pull/14251#discussion_r531718698
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java
##
@@ -932,7 +932,7 @@ private void resetAndStartScheduler() throws Exception {
);
}
- schedulerAssignedFuture.thenRun(this::startScheduling);
+
FutureUtils.assertNoException(schedulerAssignedFuture.thenRun(this::startScheduling));
Review comment:
The intention of this PR is to fix the problem of swallowed exceptions
and not the job lifecycle management based on operator coordinator failures. I
don't know whether restarting a job makes sense if the operator coordinator has
failed to start. To me it looks as if the operator coordinator integration
hasn't been completed yet. I would consider this as a follow up task.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531718518
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[GitHub] [flink] tillrohrmann commented on a change in pull request #14251: [FLINK-20382][runtime] Fail hard when JobMaster cannot start scheduling of a job
tillrohrmann commented on a change in pull request #14251: URL: https://github.com/apache/flink/pull/14251#discussion_r531717894 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/util/FatalExitExceptionHandler.java ## @@ -32,6 +32,7 @@ private static final Logger LOG = LoggerFactory.getLogger(FatalExitExceptionHandler.class); public static final FatalExitExceptionHandler INSTANCE = new FatalExitExceptionHandler(); + public static final int EXIT_CODE = -17; Review comment: I would see this an orthogonal change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
flinkbot commented on pull request #14254: URL: https://github.com/apache/flink/pull/14254#issuecomment-734938232 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9f8f123a323372be3edd985c83d8af11528dec16 (Fri Nov 27 17:55:04 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20357) Rework HA documentation page
[ https://issues.apache.org/jira/browse/FLINK-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20357: --- Labels: pull-request-available (was: ) > Rework HA documentation page > > > Key: FLINK-20357 > URL: https://issues.apache.org/jira/browse/FLINK-20357 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / Coordination >Affects Versions: 1.12.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > We need to rework the HA documentation page. The first step is to split the > existing documentation into general concepts as an overview page and HA > service implementation specific sub pages. For the implementation specific > sub pages we need to add Zookeeper and the K8s HA services. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tillrohrmann opened a new pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
tillrohrmann opened a new pull request #14254: URL: https://github.com/apache/flink/pull/14254 This commit splits the HA documentation up into a general overview and the specific implementations: * ZooKeeper HA services * Kubernetes HA services Moreover, this commit moves resource-provider specific documentation to the respective resource-provider documentation. This is done in order to not lose this information and it should be properly incorporated once the resource-provider documentation is updated. cc @rmetzger, @XComp, @wangyang0918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14247: [FLINK-20175][bp-1.11] Avro Confluent Registry SQL format does not support adding nullable columns
flinkbot edited a comment on pull request #14247: URL: https://github.com/apache/flink/pull/14247#issuecomment-734770560 ## CI report: * 70cf3a3d81f9d5499e5008c5c5eca93a4cdb0c7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10264) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531709904
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
Review comment:
I think it only makes sense to mention `HADOOP_CONF_DIR`. Afaik
`HADOOP_HOME` is from Hadoop 1.x times, and deprecated.
I will mention `HADOOP_CONF_DIR` in the reference section about the
configuration, to not confuse new users in the beginners section.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * 983518fa6cdff38b3264506a9ca54178699c9525 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10257) * b4d5fbae5b72a01ab9e94bc000373d6797339f9a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10266) * 6226047b9ce026f6431f9b8f7eed7ab49b373170 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10268) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
tillrohrmann commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531702295
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531698001
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[jira] [Commented] (FLINK-20396) Replace "OperatorCoordinator.subtaskFailed()" with "subtaskRestored()"
[
https://issues.apache.org/jira/browse/FLINK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239774#comment-17239774
]
Stephan Ewen commented on FLINK-20396:
--
In the first version, I would add the {{subtaskReset()}} method in addition to
the {{subtaskFailed()}} method because of two reasons:
- {{subtaskFailed()}} can potentially do slightly faster cleanup (for example
unregistering readers)
- It is complex to communicate a failure cause to {{subtaskReset()}}. If we
want to make
We can decide to remove {{subtaskFailed()}} in the future, if we feel we don't
need the slightly faster notification, or the failure reason.
If we want to retain the failure reason, but do not care about the slightly
faster notification, we can consolidate the two into a single method
{{subtaskReset(int subtask, long checkpointId, Throwable failureCause);}}.
The {{OperatorCoordinatorHolder}} can remember the exceptions per subtask
between failure and restore to pass them to the coordinator.
Because that needs potentially noticeably more heap memory (retain many
exceptions with stack traces) I would suggest to do that change "if needed" and
not immediately.
> Replace "OperatorCoordinator.subtaskFailed()" with "subtaskRestored()"
> --
>
> Key: FLINK-20396
> URL: https://issues.apache.org/jira/browse/FLINK-20396
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.2
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> There are no strong order guarantees between
> {{OperatorCoordinator.subtaskFailed()}} and
> {{OperatorCoordinator.notifyCheckpointComplete()}}.
> It can happen that a checkpoint completes after the notification for task
> failure is sent:
> - {{OperatorCoordinator.checkpoint()}}
> - {{OperatorCoordinator.subtaskFailed()}}
> - {{OperatorCoordinator.checkpointComplete()}}
> The subtask failure here does not know whether the previous checkpoint
> completed or not. It cannot decide what state the subtask will be in after
> recovery.
> There is no easy fix right now to strictly guarantee the order of the method
> calls, so alternatively we need to provide the necessary information to
> reason about the status of tasks.
> We should replace {{OperatorCoordinator.subtaskFailed(int subtask)}} with
> {{OperatorCoordinator.subtaskRestored(int subtask, long checkpoint)}}. That
> implementations get the explicit checkpoint ID for the subtask recovery, and
> can align that with the IDs of checkpoints that were taken.
> It is still (in rare cases) possible that for a specific checkpoint C,
> {{OperatorCoordinator.subtaskRestored(subtaskIndex, C)) comes before
> {{OperatorCoordinator.checkpointComplete(C)}}.
> h3. Background
> The Checkpointing Procedure is partially asynchronous on the {{JobManager}} /
> {{CheckpointCoordinator}}: After all subtasks acknowledged the checkpoint,
> the finalization (writing out metadata and registering the checkpoint in
> ZooKeeper) happens in an I/O thread, and the checkpoint completes after that.
> This sequence of events can happen:
> - tasks acks checkpoint
> - checkpoint fully acknowledged, finalization starts
> - task fails
> - task failure notification is dispatched
> - checkpoint completes.
> For task failures and checkpoint completion, no order is defined.
> However, for task restore and checkpoint completion, the order is well
> defined: When a task is restored, pending checkpoints are either canceled or
> complete. None can be within finalization. That is currently guaranteed with
> a lock in the {{CheckpointCoordinator}}.
> (An implication of that being that restores can be blocking operations in the
> scheduler, which is not ideal from the perspective of making the scheduler
> async/non-blocking, but it is currently essential for correctness).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20396) Replace "OperatorCoordinator.subtaskFailed()" with "subtaskRestored()"
[
https://issues.apache.org/jira/browse/FLINK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239774#comment-17239774
]
Stephan Ewen edited comment on FLINK-20396 at 11/27/20, 4:43 PM:
-
In the first version, I would add the {{subtaskReset()}} method in addition to
the {{subtaskFailed()}} method because of two reasons:
- {{subtaskFailed()}} can potentially do slightly faster cleanup (for example
unregistering readers)
- It is complex to communicate a failure cause to {{subtaskReset()}}. If we
want to make actions dependent on exception types, we need to handle that in
the {{subtaskFailed()}} method.
We can decide to remove {{subtaskFailed()}} in the future, if we feel we don't
need the slightly faster notification, or the failure reason.
If we want to retain the failure reason, but do not care about the slightly
faster notification, we can consolidate the two into a single method
{{subtaskReset(int subtask, long checkpointId, Throwable failureCause);}}.
The {{OperatorCoordinatorHolder}} can remember the exceptions per subtask
between failure and restore to pass them to the coordinator.
Because that needs potentially noticeably more heap memory (retain many
exceptions with stack traces) I would suggest to do that change "if needed" and
not immediately.
was (Author: stephanewen):
In the first version, I would add the {{subtaskReset()}} method in addition to
the {{subtaskFailed()}} method because of two reasons:
- {{subtaskFailed()}} can potentially do slightly faster cleanup (for example
unregistering readers)
- It is complex to communicate a failure cause to {{subtaskReset()}}. If we
want to make
We can decide to remove {{subtaskFailed()}} in the future, if we feel we don't
need the slightly faster notification, or the failure reason.
If we want to retain the failure reason, but do not care about the slightly
faster notification, we can consolidate the two into a single method
{{subtaskReset(int subtask, long checkpointId, Throwable failureCause);}}.
The {{OperatorCoordinatorHolder}} can remember the exceptions per subtask
between failure and restore to pass them to the coordinator.
Because that needs potentially noticeably more heap memory (retain many
exceptions with stack traces) I would suggest to do that change "if needed" and
not immediately.
> Replace "OperatorCoordinator.subtaskFailed()" with "subtaskRestored()"
> --
>
> Key: FLINK-20396
> URL: https://issues.apache.org/jira/browse/FLINK-20396
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.2
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> There are no strong order guarantees between
> {{OperatorCoordinator.subtaskFailed()}} and
> {{OperatorCoordinator.notifyCheckpointComplete()}}.
> It can happen that a checkpoint completes after the notification for task
> failure is sent:
> - {{OperatorCoordinator.checkpoint()}}
> - {{OperatorCoordinator.subtaskFailed()}}
> - {{OperatorCoordinator.checkpointComplete()}}
> The subtask failure here does not know whether the previous checkpoint
> completed or not. It cannot decide what state the subtask will be in after
> recovery.
> There is no easy fix right now to strictly guarantee the order of the method
> calls, so alternatively we need to provide the necessary information to
> reason about the status of tasks.
> We should replace {{OperatorCoordinator.subtaskFailed(int subtask)}} with
> {{OperatorCoordinator.subtaskRestored(int subtask, long checkpoint)}}. That
> implementations get the explicit checkpoint ID for the subtask recovery, and
> can align that with the IDs of checkpoints that were taken.
> It is still (in rare cases) possible that for a specific checkpoint C,
> {{OperatorCoordinator.subtaskRestored(subtaskIndex, C)) comes before
> {{OperatorCoordinator.checkpointComplete(C)}}.
> h3. Background
> The Checkpointing Procedure is partially asynchronous on the {{JobManager}} /
> {{CheckpointCoordinator}}: After all subtasks acknowledged the checkpoint,
> the finalization (writing out metadata and registering the checkpoint in
> ZooKeeper) happens in an I/O thread, and the checkpoint completes after that.
> This sequence of events can happen:
> - tasks acks checkpoint
> - checkpoint fully acknowledged, finalization starts
> - task fails
> - task failure notification is dispatched
> - checkpoint completes.
> For task failures and checkpoint completion, no order is defined.
> However, for task restore and checkpoint completion, the order is well
> defined: When a task is restored, pending checkpoints are either canceled or
> complete. None can be within
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * d97f8149d6fcb9985cd75c4cd8f30b9cd10f14e5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10233) * 983518fa6cdff38b3264506a9ca54178699c9525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10257) * b4d5fbae5b72a01ab9e94bc000373d6797339f9a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10266) * 6226047b9ce026f6431f9b8f7eed7ab49b373170 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10268) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14234: Make FileSink Support migrating from StreamingFileSink
flinkbot edited a comment on pull request #14234: URL: https://github.com/apache/flink/pull/14234#issuecomment-734232999 ## CI report: * c89575f22a01309ffb329e1d0ecdc07b3c99fc91 UNKNOWN * 3f569564dae3e5ee5a9b76ad86dc3acf9685d9df Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10255) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * d97f8149d6fcb9985cd75c4cd8f30b9cd10f14e5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10233) * 983518fa6cdff38b3264506a9ca54178699c9525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10257) * b4d5fbae5b72a01ab9e94bc000373d6797339f9a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10266) * 6226047b9ce026f6431f9b8f7eed7ab49b373170 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14157: [FLINK-19969] CLI print run-application help msg
flinkbot edited a comment on pull request #14157: URL: https://github.com/apache/flink/pull/14157#issuecomment-731272625 ## CI report: * f8b909654df226298fd5f5c21ece16ebb55d5ea1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10254) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20405) The LAG function in over window is not implements correctly
Leonard Xu created FLINK-20405:
--
Summary: The LAG function in over window is not implements
correctly
Key: FLINK-20405
URL: https://issues.apache.org/jira/browse/FLINK-20405
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.12.0
Reporter: Leonard Xu
For LAG(input, offset, default) function in over window, it always return
current row's input no matter how the offset is set.
After see the codegen code of the function, I think the implementation is not
correct and need to correct.
{code:java}
// the offset and default value is never used
public UnboundedOverAggregateHelper$24(java.lang.Object[] references) throws
Exception {constant$14 = ((int) 1);
constant$14isNull = false;
constant$15 = ((org.apache.flink.table.data.binary.BinaryStringData) str$13);
constant$15isNull = false;typeSerializer$19 =
(((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
references[0])); }
public void accumulate(org.apache.flink.table.data.RowData accInput) throws
Exception {
org.apache.flink.table.data.binary.BinaryStringData field$21;
boolean isNull$21;
org.apache.flink.table.data.binary.BinaryStringData field$22;
isNull$21 = accInput.isNullAt(2);field$21 =
org.apache.flink.table.data.binary.BinaryStringData.EMPTY_UTF8;if
(!isNull$21) { field$21 =
((org.apache.flink.table.data.binary.BinaryStringData) accInput.getString(2));
}field$22 = field$21;if (!isNull$21) {
field$22 = (org.apache.flink.table.data.binary.BinaryStringData)
(typeSerializer$19.copy(field$22));}
if (agg0_leadlag != field$22) { agg0_leadlag =
((org.apache.flink.table.data.binary.BinaryStringData)
typeSerializer$19.copy(field$22));} ;
agg0_leadlagIsNull = isNull$21; }
{code}
The question comes from user mail list[1]
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkSQL-kafka-gt-dedup-gt-kafka-td39335.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14249: [FLINK-20391] Set FORWARD_EDGES_PIPELINED for BATCH ExecutionMode
flinkbot edited a comment on pull request #14249: URL: https://github.com/apache/flink/pull/14249#issuecomment-734789081 ## CI report: * 8092920b062657c9f8fed976a7b1792411a92a59 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10252) * 2ce421de97821f5fa47a965362d3c08c4971fa92 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10260) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14250: [FLINK-20389][tests] Fix UnalignedCheckpointITCase to work with unassigned splits.
flinkbot edited a comment on pull request #14250: URL: https://github.com/apache/flink/pull/14250#issuecomment-734789670 ## CI report: * 12672b400e4237bb5ec5ada14fae4c2f6596b03d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10253) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
flinkbot edited a comment on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-732845197 ## CI report: * 8139d3ee5e5fd595279a4a4c6e7849d1a9dfc6b5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10049) * 6ffe3e161a8ec42b7ce5da3c46b57587d98a1f90 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10267) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Shawn-Hx commented on pull request #14235: [FLINK-20298][docs] Replace usage of {{ site.baseurl }} in documentation
Shawn-Hx commented on pull request #14235: URL: https://github.com/apache/flink/pull/14235#issuecomment-734892872 Thanks for the review @dawidwys I need some time to be familiar with the grammar of Jekyll. I wonder if I could open another JIRA ticket to solve the remaining problems because this PR has changed a large number of files, and if others change these files there might be lots of conflicts to resolve. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531669932
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[jira] [Updated] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
[
https://issues.apache.org/jira/browse/FLINK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-20404:
-
Priority: Blocker (was: Major)
> ZooKeeper quorum fails to start due to missing log4j library
>
>
> Key: FLINK-20404
> URL: https://issues.apache.org/jira/browse/FLINK-20404
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.11.2
>Reporter: Pedro Miguel Rainho Chaves
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
> throws the following exception.
>
> {code:java}
> 2020-11-27 13:13:38,371 ERROR
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error
> running ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
> java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
> ~[flink-dist_2.11-1.11.2.jar:1.11.2]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
> [flink-dist_2.11-1.11.2.jar:1.11.2]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.log4j.jmx.HierarchyDynamicMBean
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> ~[?:1.8.0_262]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ~[?:1.8.0_262]
> ... 4 more
> {code}
> This happens because the new flink version is missing a log4j library. This
> can be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
> bootstrapped zookeepeer version should be compatible with the log4j2
> libraries that come with flink's default installation.
>
> *Steps to reproduce:*
> # Fresh install of flink version 1.11.2
> # Change the zookeeper config to start as a quorum
> {code:java}
> server.1=:2888:3888
> server.2=:2888:3888{code}
> # Start zookeeper
> # /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
[
https://issues.apache.org/jira/browse/FLINK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-20404:
-
Fix Version/s: 1.11.3
1.12.0
> ZooKeeper quorum fails to start due to missing log4j library
>
>
> Key: FLINK-20404
> URL: https://issues.apache.org/jira/browse/FLINK-20404
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.11.2
>Reporter: Pedro Miguel Rainho Chaves
>Priority: Major
> Fix For: 1.12.0, 1.11.3
>
>
> Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
> throws the following exception.
>
> {code:java}
> 2020-11-27 13:13:38,371 ERROR
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error
> running ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
> java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
> ~[flink-dist_2.11-1.11.2.jar:1.11.2]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
> [flink-dist_2.11-1.11.2.jar:1.11.2]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.log4j.jmx.HierarchyDynamicMBean
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> ~[?:1.8.0_262]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ~[?:1.8.0_262]
> ... 4 more
> {code}
> This happens because the new flink version is missing a log4j library. This
> can be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
> bootstrapped zookeepeer version should be compatible with the log4j2
> libraries that come with flink's default installation.
>
> *Steps to reproduce:*
> # Fresh install of flink version 1.11.2
> # Change the zookeeper config to start as a quorum
> {code:java}
> server.1=:2888:3888
> server.2=:2888:3888{code}
> # Start zookeeper
> # /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531666867
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[GitHub] [flink] XComp commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
XComp commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r531663140
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
we can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by the specified remote locations rather than be
shipped to the cluster by the
+client.
+
+### Per-Job Cluster Mode
+
+The Per-job Cluster mode will launch a Flink cluster on YARN, then run the
provided application jar locally and finally submit the JobGraph to the
JobManager on YARN. If you pass the `--detached` argument, the client will stop
once the submission is accepted.
+
+The YARN cluster will stop once the job has stopped.
+
+{% highlight bash %}
+./bin/flink run -m
[GitHub] [flink] flinkbot edited a comment on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
flinkbot edited a comment on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-732845197 ## CI report: * 8139d3ee5e5fd595279a4a4c6e7849d1a9dfc6b5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10049) * 6ffe3e161a8ec42b7ce5da3c46b57587d98a1f90 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * d97f8149d6fcb9985cd75c4cd8f30b9cd10f14e5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10233) * 983518fa6cdff38b3264506a9ca54178699c9525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10257) * b4d5fbae5b72a01ab9e94bc000373d6797339f9a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10266) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin commented on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-734883917 I added TM failure for batch case. The JM failure is tricky w/o synchronization with the job execution. The job gets done before JM failure. We could make `EmbeddedHaServices::getJobManagerLeaderService` public to see whether the JM is still available to restart but this is fragile to concurrency. Hence, I am not sure about the value of this test. I added JM failure test for batch with the job execution synchronization, abstracting it away, in the last commit. I can remove the last commit if it looks too complicated and there is no better idea how to do the JM failure test for batch in a simpler way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
Pedro Miguel Rainho Chaves created FLINK-20404:
--
Summary: ZooKeeper quorum fails to start due to missing log4j
library
Key: FLINK-20404
URL: https://issues.apache.org/jira/browse/FLINK-20404
Project: Flink
Issue Type: Bug
Affects Versions: 1.11.2
Reporter: Pedro Miguel Rainho Chaves
Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
throws the following exception.
{code:java}
2020-11-27 13:13:38,371 ERROR
org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error running
ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
at
org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
~[flink-dist_2.11-1.11.2.jar:1.11.2]
at
org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
[flink-dist_2.11-1.11.2.jar:1.11.2]
Caused by: java.lang.ClassNotFoundException:
org.apache.log4j.jmx.HierarchyDynamicMBean
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_262]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_262]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
~[?:1.8.0_262]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_262]
... 4 more
{code}
This happens because the new flink version is missing a log4j library. This can
be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
bootstrapped zookeepeer version should be compatible with the log4j2 libraries
that come with flink's default installation.
*Steps to reproduce:*
# Fresh install of flink version 1.11.2
# Change the zookeeper config to start as a quorum
{code:java}
server.1=:2888:3888
server.2=:2888:3888{code}
# Start zookeeper
# /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14234: Make FileSink Support migrating from StreamingFileSink
flinkbot edited a comment on pull request #14234: URL: https://github.com/apache/flink/pull/14234#issuecomment-734232999 ## CI report: * b461ede54e0564d0b1fcab49aef07c7b9a3c107a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10247) * c89575f22a01309ffb329e1d0ecdc07b3c99fc91 UNKNOWN * 3f569564dae3e5ee5a9b76ad86dc3acf9685d9df Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10255) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14238: [FLINK-20347] Rework YARN documentation page
flinkbot edited a comment on pull request #14238: URL: https://github.com/apache/flink/pull/14238#issuecomment-734336545 ## CI report: * 7806bb7e0dbca1263ddffb771dec367807f0590a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10261) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14204: [FLINK-20325][build] Move docs_404_check to CI stage
flinkbot edited a comment on pull request #14204: URL: https://github.com/apache/flink/pull/14204#issuecomment-733048767 ## CI report: * 6550d3e1b01af94d4f652f993834b75272da1020 UNKNOWN * 38ed086f80ee2a94613a716d090574beed92aef4 UNKNOWN * d97f8149d6fcb9985cd75c4cd8f30b9cd10f14e5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10233) * 983518fa6cdff38b3264506a9ca54178699c9525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10257) * b4d5fbae5b72a01ab9e94bc000373d6797339f9a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-19852) Managed memory released check can block IterativeTask
[ https://issues.apache.org/jira/browse/FLINK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-19852. - Resolution: Fixed > Managed memory released check can block IterativeTask > - > > Key: FLINK-19852 > URL: https://issues.apache.org/jira/browse/FLINK-19852 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.11.0, 1.10.2, 1.12.0, 1.11.1, 1.11.2 >Reporter: shaomeng.wang >Assignee: Roman Khachatryan >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0, 1.11.3 > > Attachments: image-2020-10-28-17-48-28-395.png, > image-2020-10-28-17-48-48-583.png > > > UnsafeMemoryBudget#reserveMemory, called on TempBarrier, needs time to wait > on GC of all allocated/released managed memory at every iteration. > > stack: > !image-2020-10-28-17-48-48-583.png! > new TempBarrier in BatchTask > !image-2020-10-28-17-48-28-395.png! > > These will be very slow than before. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19852) Managed memory released check can block IterativeTask
[ https://issues.apache.org/jira/browse/FLINK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239718#comment-17239718 ] Arvid Heise commented on FLINK-19852: - Merged into 1.11 as 05a7875448b6f92450083c17153e7f21e7de2e31. > Managed memory released check can block IterativeTask > - > > Key: FLINK-19852 > URL: https://issues.apache.org/jira/browse/FLINK-19852 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.11.0, 1.10.2, 1.12.0, 1.11.1, 1.11.2 >Reporter: shaomeng.wang >Assignee: Roman Khachatryan >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0, 1.11.3 > > Attachments: image-2020-10-28-17-48-28-395.png, > image-2020-10-28-17-48-48-583.png > > > UnsafeMemoryBudget#reserveMemory, called on TempBarrier, needs time to wait > on GC of all allocated/released managed memory at every iteration. > > stack: > !image-2020-10-28-17-48-48-583.png! > new TempBarrier in BatchTask > !image-2020-10-28-17-48-28-395.png! > > These will be very slow than before. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #14221: (1.11) [FLINK-19852][task] Reuse TempBarrier memory between iterations
AHeise merged pull request #14221: URL: https://github.com/apache/flink/pull/14221 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239716#comment-17239716 ] Dawid Wysakowicz edited comment on FLINK-20392 at 11/27/20, 2:57 PM: - Just to add to the FLINK-19997, I set up a FlinkContainer that runs a Flink cluster within a docker container with a subset of functionalities provided by the FlinkResource. It can be a good base to move the FlinkResource onto testcontainers entirely. was (Author: dawidwys): Just to add to the FLINK-19997, I set up a FlinkContainer that runs a Flink cluster within a docker container with a subset of functionalities provided by the FlinkResource. It can be a good base to move the FlinkResource onto testcontainers. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Test > Components: Test Infrastructure, Tests >Reporter: Matthias >Priority: Major > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
