[GitHub] [flink] klion26 commented on a change in pull request #13410: [FLINK-19247][docs-zh] Update Chinese documentation after removal of Kafka 0.10 and 0.11
klion26 commented on a change in pull request #13410:
URL: https://github.com/apache/flink/pull/13410#discussion_r496433356
##
File path: docs/dev/connectors/kafka.zh.md
##
@@ -431,39 +371,46 @@ stream.addSink(myProducer);
{% highlight scala %}
val stream: DataStream[String] = ...
+Properties properties = new Properties
+properties.setProperty("bootstrap.servers", "localhost:9092")
+
val myProducer = new FlinkKafkaProducer[String](
-"localhost:9092", // broker 列表
"my-topic", // 目标 topic
-new SimpleStringSchema) // 序列化 schema
-
-// 0.10+ 版本的 Kafka 允许在将记录写入 Kafka 时附加记录的事件时间戳；
-// 此方法不适用于早期版本的 Kafka
-myProducer.setWriteTimestampToKafka(true)
+new SimpleStringSchema(), // 序列化 schema
+properties, // producer 配置
+FlinkKafkaProducer.Semantic.EXACTLY_ONCE) // 容错
stream.addSink(myProducer)
{% endhighlight %}
-上面的例子演示了创建 Flink Kafka Producer 来将流消息写入单个 Kafka 目标 topic 的基本用法。
-对于更高级的用法，这还有其他构造函数变体允许提供以下内容：
+
+
+## `SerializationSchema`
- * *提供自定义属性*：producer 允许为内部 `KafkaProducer` 提供自定义属性配置。有关如何配置 Kafka Producer
的详细信息，请参阅 [Apache Kafka 文档](https://kafka.apache.org/documentation.html)。
- * *自定义分区器*：要将消息分配给特定的分区，可以向构造函数提供一个 `FlinkKafkaPartitioner`
的实现。这个分区器将被流中的每条记录调用，以确定消息应该发送到目标 topic 的哪个具体分区里。有关详细信息，请参阅 [Kafka Producer
分区方案](#kafka-producer-分区方案)。
- * *高级的序列化 schema*：与 consumer 类似，producer 还允许使用名为 `KeyedSerializationSchema`
的高级序列化 schema，该 schema 允许单独序列化 key 和 value。它还允许覆盖目标 topic，以便 producer
实例可以将数据发送到多个 topic。
+Flink Kafka Producer 需要知道如何将 Java/Scala 对象转化为二进制数据。
-### Kafka Producer 分区方案
+`KafkaSerializationSchema` 允许用户指定这样的 schema。它会为每个记录调用 `ProducerRecord serialize(T element, @Nullable Long timestamp)` 方法，产生一个写入到 Kafka 的
`ProducerRecord`。
-默认情况下，如果没有为 Flink Kafka Producer 指定自定义分区程序，则 producer 将使用
`FlinkFixedPartitioner` 为每个 Flink Kafka Producer 并行子任务映射到单个 Kafka
分区（即，接收子任务接收到的所有消息都将位于同一个 Kafka 分区中）。
+用户可以对如何将数据写到 Kafka 进行细粒度的控制。你可以通过 producer record：
-可以通过扩展 `FlinkKafkaPartitioner` 类来实现自定义分区程序。所有 Kafka 版本的构造函数都允许在实例化 producer
时提供自定义分区程序。
-注意：分区器实现必须是可序列化的，因为它们将在 Flink 节点之间传输。此外，请记住分区器中的任何状态都将在作业失败时丢失，因为分区器不是
producer 的 checkpoint 状态的一部分。
+* 设置 header 值
+* 为每个 record 定义 key
+* 指定数据的自定义分区
-也可以完全避免使用分区器，并简单地让 Kafka 通过其附加 key 写入的消息进行分区（使用提供的序列化 schema 为每条记录确定分区）。
-为此，在实例化 producer 时提供 `null` 自定义分区程序，提供 `null` 作为自定义分区器是很重要的;
如上所述，如果未指定自定义分区程序，则默认使用 `FlinkFixedPartitioner`。
+
### Kafka Producer 和容错
+启用 Flink 的 checkpointing 后，`FlinkKafkaProducer` 可以提供精确一次的语义保证。
+
+除了启用 Flink 的 checkpointing，你也可以通过将适当的 `semantic` 参数传递给 `FlinkKafkaProducer`
来选择三种不同的操作模式：
+
+* `Semantic.NONE`：Flink 不会有任何语义的保证，产生的记录可能会丢失或重复。
+* `Semantic.AT_LEAST_ONCE`（默认设置）：可以保证不会丢失任何记录（但是记录可能会重复）
+* `Semantic.EXACTLY_ONCE`：使用 Kafka 事务提供精确一次语义。无论何时，在使用事务写入 Kafka 时，都要记得为所有消费
Kafka 消息的应用程序设置所需的 `isolation.level`（`read_committed` 或 `read_uncommitted` -
后者是默认值）。
+
# 注意事项
Review comment:
这里是不是漏了 标签
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13482: test
flinkbot edited a comment on pull request #13482: URL: https://github.com/apache/flink/pull/13482#issuecomment-698903502 ## CI report: * b6d1f311cb2a041680b89e3aedfe3d5cbdca42d0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7034) * dc45ef6deb27f17c6a988b4ae290cd35ffc1645f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * 7a2be06f74b2abc1d40f345fe995634e0f246413 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7046) * d7638e715ec7af65f8a45e276204ce86bce4369c UNKNOWN * cda95f3a0ee6988a3852d5e731ad360484bad53a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7050) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13487: [FLINK-19409][Documentation] remove unrelated comment for getValue in ListView
flinkbot edited a comment on pull request #13487: URL: https://github.com/apache/flink/pull/13487#issuecomment-699575614 ## CI report: * 9cbc32fe3fb52c05dce29124ac61c4a7c2952241 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6985) * f756ebd3f2f727c01f63e63e27f5fb7be2c8120f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7053) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13500: [FLINK-19180][state backends] Make RocksDB respect managed memory fraction
flinkbot edited a comment on pull request #13500: URL: https://github.com/apache/flink/pull/13500#issuecomment-699908639 ## CI report: * bee5be24b69942ac4b6323da5bd701e8287f145b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7035) * 832f7d5dd81f18d65e1a9b616e48bffc037221fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7052) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13487: [FLINK-19409][Documentation] remove unrelated comment for getValue in ListView
flinkbot edited a comment on pull request #13487: URL: https://github.com/apache/flink/pull/13487#issuecomment-699575614 ## CI report: * 9cbc32fe3fb52c05dce29124ac61c4a7c2952241 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6985) * f756ebd3f2f727c01f63e63e27f5fb7be2c8120f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19443) runtime function 'splitIndex'
[
https://issues.apache.org/jira/browse/FLINK-19443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203628#comment-17203628
]
Benchao Li commented on FLINK-19443:
[~mytang0] Thanks for reporting this, would you like to fix this?
> runtime function 'splitIndex'
> -
>
> Key: FLINK-19443
> URL: https://issues.apache.org/jira/browse/FLINK-19443
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: mytang0
>Priority: Minor
>
> runtime function 'splitIndex' has NPE problem (located in the
> SqlFunctionUtils class)
>
> *NPE version:*
> public static String splitIndex(String str, String separator, int index) {
> if (index < 0) {
> return null;
> }
> String[] values = StringUtils.splitByWholeSeparatorPreserveAllTokens(str,
> separator);
> if (index >= values.length) {
> return null;
> } else {
> return values[index];
> }
> }
> public static String splitIndex(String str, int character, int index) {
> if (character > 255 || character < 1 || index < 0) {
> return null;
> }
> String[] values = StringUtils.splitPreserveAllTokens(str, (char)
> character);
> if (index >= values.length) {
> return null;
> } else {
> return values[index];
> }
> }
>
> *Fix version:*
> public static String splitIndex(String str, String separator, int index) {
> if (index < 0) {
> return null;
> }
> String[] values = StringUtils.splitByWholeSeparatorPreserveAllTokens(str,
> separator);
> if ({color:#FF}values == null ||{color} index >= values.length) {
> return null;
> } else {
> return values[index];
> }
> }
> public static String splitIndex(String str, int character, int index) {
> if (character > 255 || character < 1 || index < 0) {
> return null;
> }
> String[] values = StringUtils.splitPreserveAllTokens(str, (char)
> character);
> if ({color:#FF}values == null ||{color} index >= values.length) {
> return null;
> } else {
> return values[index];
> }
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13500: [FLINK-19180][state backends] Make RocksDB respect managed memory fraction
flinkbot edited a comment on pull request #13500: URL: https://github.com/apache/flink/pull/13500#issuecomment-699908639 ## CI report: * bee5be24b69942ac4b6323da5bd701e8287f145b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7035) * 832f7d5dd81f18d65e1a9b616e48bffc037221fa UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * 921b49a2ccb91a91ff0f3c3cf176fec0966459ff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7044) * c1ead553fe6892d93e5fc702f51d3aff397d7b78 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7047) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13356: [FLINK-16789][runtime][rest] Enable JMX RMI port retrieval via REST API
flinkbot edited a comment on pull request #13356: URL: https://github.com/apache/flink/pull/13356#issuecomment-688955548 ## CI report: * c95d0e33b9cd23264c899aae7df75c83d290f129 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7007) * 174ea945be035310bd0f27bc3d00af5c815cda3d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7051) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19386) Remove legacy table planner dependecy from connectors and formats
[ https://issues.apache.org/jira/browse/FLINK-19386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203624#comment-17203624 ] hailong wang edited comment on FLINK-19386 at 9/29/20, 4:23 AM: [~jark], here[1] is my idea. can I go ahead with it? [1][lRemove dependency of legacy planner from Connectors and Formats|https://docs.google.com/document/d/1XsUV9WFZpRkm6as4irIV-tHKyEnU7lr6GR8V9vLSNjI/edit?pli=1#heading=h.sfk98udc1vm7] was (Author: hailong wang): [~jark], here[1] is my idea. can I go ahead with it? [1][https://docs.google.com/document/d/1XsUV9WFZpRkm6as4irIV-tHKyEnU7lr6GR8V9vLSNjI/edit?pli=1#heading=h.sfk98udc1vm7|http://example.com] > Remove legacy table planner dependecy from connectors and formats > - > > Key: FLINK-19386 > URL: https://issues.apache.org/jira/browse/FLINK-19386 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner >Affects Versions: 1.11.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Major > Fix For: 1.12.0 > > > I found some connectors denpendecy legacy table planner and used in test > class. > For example `HiveCatalogUseBlinkITCase ` use `JavaScalaConversionUtil` in > legacy table planner not blink planner. For FLIP-32 has be accepted, I think > we can remove dependecy of legacy table planner from connectors and formats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19386) Remove legacy table planner dependecy from connectors and formats
[ https://issues.apache.org/jira/browse/FLINK-19386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203624#comment-17203624 ] hailong wang commented on FLINK-19386: -- [~jark], here[1] is my idea. can I go ahead with it? [1][https://docs.google.com/document/d/1XsUV9WFZpRkm6as4irIV-tHKyEnU7lr6GR8V9vLSNjI/edit?pli=1#heading=h.sfk98udc1vm7|http://example.com] > Remove legacy table planner dependecy from connectors and formats > - > > Key: FLINK-19386 > URL: https://issues.apache.org/jira/browse/FLINK-19386 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner >Affects Versions: 1.11.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Major > Fix For: 1.12.0 > > > I found some connectors denpendecy legacy table planner and used in test > class. > For example `HiveCatalogUseBlinkITCase ` use `JavaScalaConversionUtil` in > legacy table planner not blink planner. For FLIP-32 has be accepted, I think > we can remove dependecy of legacy table planner from connectors and formats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) * 7a2be06f74b2abc1d40f345fe995634e0f246413 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7046) * d7638e715ec7af65f8a45e276204ce86bce4369c UNKNOWN * cda95f3a0ee6988a3852d5e731ad360484bad53a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7050) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13356: [FLINK-16789][runtime][rest] Enable JMX RMI port retrieval via REST API
flinkbot edited a comment on pull request #13356: URL: https://github.com/apache/flink/pull/13356#issuecomment-688955548 ## CI report: * c95d0e33b9cd23264c899aae7df75c83d290f129 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7007) * 174ea945be035310bd0f27bc3d00af5c815cda3d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) * 7a2be06f74b2abc1d40f345fe995634e0f246413 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7046) * d7638e715ec7af65f8a45e276204ce86bce4369c UNKNOWN * cda95f3a0ee6988a3852d5e731ad360484bad53a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19419) "null-string-literal" does not work in HBaseSource decoder
[ https://issues.apache.org/jira/browse/FLINK-19419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CaoZhen closed FLINK-19419. --- Resolution: Not A Problem > "null-string-literal" does not work in HBaseSource decoder > > > Key: FLINK-19419 > URL: https://issues.apache.org/jira/browse/FLINK-19419 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Affects Versions: 1.11.0, 1.11.1, 1.11.2 >Reporter: CaoZhen >Priority: Minor > Labels: pull-request-available > Attachments: image-2020-09-25-21-11-36-418.png > > > > When using HBaseSoucre, it is found that "null-string-literal" does not work. > The current decoder processing logic is shown below. > `nullStringBytes` should be used when the `value` is null. > > !image-2020-09-25-21-11-36-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19419) "null-string-literal" does not work in HBaseSource decoder
[ https://issues.apache.org/jira/browse/FLINK-19419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203621#comment-17203621 ] CaoZhen commented on FLINK-19419: - [~jark] Thanks for your explanation. > "null-string-literal" does not work in HBaseSource decoder > > > Key: FLINK-19419 > URL: https://issues.apache.org/jira/browse/FLINK-19419 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Affects Versions: 1.11.0, 1.11.1, 1.11.2 >Reporter: CaoZhen >Priority: Minor > Labels: pull-request-available > Attachments: image-2020-09-25-21-11-36-418.png > > > > When using HBaseSoucre, it is found that "null-string-literal" does not work. > The current decoder processing logic is shown below. > `nullStringBytes` should be used when the `value` is null. > > !image-2020-09-25-21-11-36-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #13489: [hotfix][docs]Fix the FROM_UNIXTIME UTC sample time error
wuchong closed pull request #13489: URL: https://github.com/apache/flink/pull/13489 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #13489: [hotfix][docs]Fix the FROM_UNIXTIME UTC sample time error
wuchong commented on pull request #13489: URL: https://github.com/apache/flink/pull/13489#issuecomment-700410808 Has been fixed in 83b429bbc7fde4b958cfc6d6726fbbe6f590. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19443) runtime function 'splitIndex'
mytang0 created FLINK-19443:
---
Summary: runtime function 'splitIndex'
Key: FLINK-19443
URL: https://issues.apache.org/jira/browse/FLINK-19443
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Reporter: mytang0
runtime function 'splitIndex' has NPE problem (located in the SqlFunctionUtils
class)
*NPE version:*
public static String splitIndex(String str, String separator, int index) {
if (index < 0) {
return null;
}
String[] values = StringUtils.splitByWholeSeparatorPreserveAllTokens(str,
separator);
if (index >= values.length) {
return null;
} else {
return values[index];
}
}
public static String splitIndex(String str, int character, int index) {
if (character > 255 || character < 1 || index < 0) {
return null;
}
String[] values = StringUtils.splitPreserveAllTokens(str, (char) character);
if (index >= values.length) {
return null;
} else {
return values[index];
}
}
*Fix version:*
public static String splitIndex(String str, String separator, int index) {
if (index < 0) {
return null;
}
String[] values = StringUtils.splitByWholeSeparatorPreserveAllTokens(str,
separator);
if ({color:#FF}values == null ||{color} index >= values.length) {
return null;
} else {
return values[index];
}
}
public static String splitIndex(String str, int character, int index) {
if (character > 255 || character < 1 || index < 0) {
return null;
}
String[] values = StringUtils.splitPreserveAllTokens(str, (char) character);
if ({color:#FF}values == null ||{color} index >= values.length) {
return null;
} else {
return values[index];
}
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19419) "null-string-literal" does not work in HBaseSource decoder
[
https://issues.apache.org/jira/browse/FLINK-19419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203619#comment-17203619
]
Jark Wu commented on FLINK-19419:
-
[~caozhen1937], we HAVE TO use an specific string literal to represent
{{null}}, this is controlled by {{null-string-literal}}. If users define
"null-string-literal=null", that means users are aware of it, then we have to
return {{null}} instead of {{"null"}} string when reading a "null" string. If
users want to keep the "null" string, he/she can change the config of
{{null-string-literal=null}}.
> "null-string-literal" does not work in HBaseSource decoder
>
>
> Key: FLINK-19419
> URL: https://issues.apache.org/jira/browse/FLINK-19419
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.11.0, 1.11.1, 1.11.2
>Reporter: CaoZhen
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2020-09-25-21-11-36-418.png
>
>
>
> When using HBaseSoucre, it is found that "null-string-literal" does not work.
> The current decoder processing logic is shown below.
> `nullStringBytes` should be used when the `value` is null.
>
> !image-2020-09-25-21-11-36-418.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18830) JoinCoGroupFunction and FlatJoinCoGroupFunction work incorrectly for outer join when one side of coGroup is empty
[
https://issues.apache.org/jira/browse/FLINK-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203617#comment-17203617
]
Jark Wu edited comment on FLINK-18830 at 9/29/20, 3:48 AM:
---
I agree with [~aljoscha]. I'm pretty sure the current window join in DataStream
API can't satisfy the Table/SQL's needs in the terms of functinality and
performance. That means we may have to have a customized implementation.
was (Author: jark):
I agree with [~aljoscha]. I'm pretty sure the current window join in DataStream
API can't satisfy the Table/SQL's needs in the terms of functinality and
performance. That means we may need to have an customized implementation
anyway.
> JoinCoGroupFunction and FlatJoinCoGroupFunction work incorrectly for outer
> join when one side of coGroup is empty
> -
>
> Key: FLINK-18830
> URL: https://issues.apache.org/jira/browse/FLINK-18830
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.1
>Reporter: liupengcheng
>Priority: Major
>
> Currently, The {{JoinCoGroupFunction}} and {{FlatJoinCoGroupFunction}} in
> JoinedStreams doesn't respect the join type, it's been implemented as doing
> join within a two-level loop. However, this is incorrect for outer join when
> one side of the coGroup is empty.
> {code}
> public void coGroup(Iterable first, Iterable second,
> Collector out) throws Exception {
> for (T1 val1: first) {
> for (T2 val2: second) {
> wrappedFunction.join(val1, val2, out);
> }
> }
> }
> {code}
> The above code is the current implementation, suppose the first input is
> non-empty, and the second input is an empty iterator, then the join
> function(`wrappedFunction`) will never be called. This will cause no data to
> be emitted for a left outer join.
> So I propose to consider join type here, and handle this case, e.g., for left
> outer join, we can emit record with right side set to null here if the right
> side is empty or can not find any match in the right side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18830) JoinCoGroupFunction and FlatJoinCoGroupFunction work incorrectly for outer join when one side of coGroup is empty
[
https://issues.apache.org/jira/browse/FLINK-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203617#comment-17203617
]
Jark Wu commented on FLINK-18830:
-
I agree with [~aljoscha]. I'm pretty sure the current window join in DataStream
API can't satisfy the Table/SQL's needs in the terms of functinality and
performance. That means we may need to have an customized implementation
anyway.
> JoinCoGroupFunction and FlatJoinCoGroupFunction work incorrectly for outer
> join when one side of coGroup is empty
> -
>
> Key: FLINK-18830
> URL: https://issues.apache.org/jira/browse/FLINK-18830
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.1
>Reporter: liupengcheng
>Priority: Major
>
> Currently, The {{JoinCoGroupFunction}} and {{FlatJoinCoGroupFunction}} in
> JoinedStreams doesn't respect the join type, it's been implemented as doing
> join within a two-level loop. However, this is incorrect for outer join when
> one side of the coGroup is empty.
> {code}
> public void coGroup(Iterable first, Iterable second,
> Collector out) throws Exception {
> for (T1 val1: first) {
> for (T2 val2: second) {
> wrappedFunction.join(val1, val2, out);
> }
> }
> }
> {code}
> The above code is the current implementation, suppose the first input is
> non-empty, and the second input is an empty iterator, then the join
> function(`wrappedFunction`) will never be called. This will cause no data to
> be emitted for a left outer join.
> So I propose to consider join type here, and handle this case, e.g., for left
> outer join, we can emit record with right side set to null here if the right
> side is empty or can not find any match in the right side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19435) jdbc JDBCOutputFormat open function invoke Class.forName(drivername)
[
https://issues.apache.org/jira/browse/FLINK-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203615#comment-17203615
]
Jark Wu commented on FLINK-19435:
-
You mean {{Class.forName()}} is not thread safe?
> jdbc JDBCOutputFormat open function invoke Class.forName(drivername)
>
>
> Key: FLINK-19435
> URL: https://issues.apache.org/jira/browse/FLINK-19435
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Affects Versions: 1.10.2
>Reporter: xiaodao
>Priority: Major
> Fix For: 1.10.3
>
>
> when we sink data to multi jdbc outputformat ,
> ```
> protected void establishConnection() throws SQLException,
> ClassNotFoundException {
> Class.forName(drivername);
> if (username == null) {
> connection = DriverManager.getConnection(dbURL);
> } else {
> connection = DriverManager.getConnection(dbURL, username, password);
> }
> }
> ```
> may cause jdbc driver deadlock. it need to change to synchronized function.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) * 7a2be06f74b2abc1d40f345fe995634e0f246413 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7046) * d7638e715ec7af65f8a45e276204ce86bce4369c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-19431) Translate "Monitoring REST API" page of "Debugging & Monitoring" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-19431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-19431:
---
Assignee: weizheng
> Translate "Monitoring REST API" page of "Debugging & Monitoring" into Chinese
> -
>
> Key: FLINK-19431
> URL: https://issues.apache.org/jira/browse/FLINK-19431
> Project: Flink
> Issue Type: Improvement
> Components: chinese-translation, Documentation
>Reporter: weizheng
>Assignee: weizheng
>Priority: Major
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/monitoring/rest_api.html
> The markdown file is located in {{flink/docs/monitoring/rest_api.zh.md}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19432) Whether to capture the updates which don't change any monitored columns
[
https://issues.apache.org/jira/browse/FLINK-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203612#comment-17203612
]
Jark Wu commented on FLINK-19432:
-
I'm +1 to this. This can be an optimization when users are aware of this. We
can introduce an option such as {{canal-json.capture-unchanged-updates}} which
is {{ture}} by default.
> Whether to capture the updates which don't change any monitored columns
> ---
>
> Key: FLINK-19432
> URL: https://issues.apache.org/jira/browse/FLINK-19432
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.11.1
>Reporter: Zhengchao Shi
>Priority: Major
> Fix For: 1.12.0
>
>
> with `debezium-json` and `canal-json`:
> Whether to capture the updates which don't change any monitored columns. This
> may happen if the monitored columns (columns defined in Flink SQL DDL) is a
> subset of the columns in database table. We can provide an optional option，
> default 'true', which means all the updates will be captured. You can set to
> 'false' to only capture changed updates
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19433) An Error example of FROM_UNIXTIME function in document
[
https://issues.apache.org/jira/browse/FLINK-19433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-19433.
---
Fix Version/s: 1.12.0
Resolution: Fixed
Fixed in master : 83b429bbc7fde4b958cfc6d6726fbbe6f590
> An Error example of FROM_UNIXTIME function in document
> --
>
> Key: FLINK-19433
> URL: https://issues.apache.org/jira/browse/FLINK-19433
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Reporter: Kyle Zhang
>Assignee: Kyle Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> In the
> documentation:[https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/functions/systemFunctions.html#temporal-functions]
> There is an example in FROM_UNIXTIME function
> {code:java}
> E.g., FROM_UNIXTIME(44) returns '1970-01-01 09:00:44' if in UTC time zone,
> but returns '1970-01-01 09:00:44' if in 'Asia/Tokyo' time zone.
> {code}
> However, the correct result should be 1970-01-01 00:00:44 in UTC time zone
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #13493: [FLINK-19433] [docs] Correct example of FROM_UNIXTIME function in document
wuchong merged pull request #13493: URL: https://github.com/apache/flink/pull/13493 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
flinkbot edited a comment on pull request #13473: URL: https://github.com/apache/flink/pull/13473#issuecomment-698203737 ## CI report: * e74c3f4d35222f876456134e4d800611da09c6cf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7029) * ef6232c564a071236900274c014325dcb24c9e97 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7048) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) * 7a2be06f74b2abc1d40f345fe995634e0f246413 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7046) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * ad9145cd549c242e40862c8ce21cf52b06927f00 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7003) * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * 921b49a2ccb91a91ff0f3c3cf176fec0966459ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7044) * c1ead553fe6892d93e5fc702f51d3aff397d7b78 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7047) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
flinkbot edited a comment on pull request #13473: URL: https://github.com/apache/flink/pull/13473#issuecomment-698203737 ## CI report: * e74c3f4d35222f876456134e4d800611da09c6cf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7029) * ef6232c564a071236900274c014325dcb24c9e97 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * b5ac11907cbbe41f2b13e2b738458a3315d61663 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6993) * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) * 7a2be06f74b2abc1d40f345fe995634e0f246413 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7046) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-10644) Batch Job: Speculative execution
[ https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203607#comment-17203607 ] fa zheng commented on FLINK-10644: -- Any progress on this issue? It will be helpful in batch work. > Batch Job: Speculative execution > > > Key: FLINK-10644 > URL: https://issues.apache.org/jira/browse/FLINK-10644 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Reporter: JIN SUN >Assignee: BoWang >Priority: Major > > Strugglers/outlier are tasks that run slower than most of the all tasks in a > Batch Job, this somehow impact job latency, as pretty much this straggler > will be in the critical path of the job and become as the bottleneck. > Tasks may be slow for various reasons, including hardware degradation, or > software mis-configuration, or noise neighboring. It's hard for JM to predict > the runtime. > To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark > has *_speculative execution_*. Speculative execution is a health-check > procedure that checks for tasks to be speculated, i.e. running slower in a > ExecutionJobVertex than the median of all successfully completed tasks in > that EJV, Such slow tasks will be re-submitted to another TM. It will not > stop the slow tasks, but run a new copy in parallel. And will kill the others > if one of them complete. > This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be > append later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] caozhen1937 commented on pull request #13296: [FLINK-18774][format][debezium] Support debezium-avro format
caozhen1937 commented on pull request #13296: URL: https://github.com/apache/flink/pull/13296#issuecomment-700393087 @wuchong 1. the 3 debezium binary data file are generated by debezium connect 2. I think it is not easy to generate one file just like debezium-data-schema-exclude.txt. Avro format data cannot be read by row, so it is impossible to read each row of data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * b5ac11907cbbe41f2b13e2b738458a3315d61663 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6993) * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) * 7a2be06f74b2abc1d40f345fe995634e0f246413 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] caozhen1937 commented on a change in pull request #13296: [FLINK-18774][format][debezium] Support debezium-avro format
caozhen1937 commented on a change in pull request #13296:
URL: https://github.com/apache/flink/pull/13296#discussion_r496344141
##
File path:
flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/debezium/DebeziumAvroFormatFactory.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.registry.confluent.debezium;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.types.RowKind;
+
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.flink.formats.avro.registry.confluent.RegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+import static
org.apache.flink.formats.avro.registry.confluent.RegistryAvroOptions.SCHEMA_REGISTRY_URL;
+
+/**
+ * Format factory for providing configured instances of Debezium Avro to
RowData {@link DeserializationSchema}.
+ */
+public class DebeziumAvroFormatFactory implements
DeserializationFormatFactory, SerializationFormatFactory {
+
+ public static final String IDENTIFIER = "debezium-avro-confluent";
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public DecodingFormat>
createDecodingFormat(
+ DynamicTableFactory.Context context,
Review comment:
checkstyle done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] caozhen1937 commented on a change in pull request #13296: [FLINK-18774][format][debezium] Support debezium-avro format
caozhen1937 commented on a change in pull request #13296:
URL: https://github.com/apache/flink/pull/13296#discussion_r496341421
##
File path:
flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/debezium/DebeziumAvroDeserializationSchema.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.registry.confluent.debezium;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import
org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroDeserializationSchema;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.Objects;
+
+import static java.lang.String.format;
+import static
org.apache.flink.table.types.utils.TypeConversions.fromLogicalToDataType;
+
+/**
+ * Deserialization schema from Debezium Avro to Flink Table/SQL internal data
structure {@link RowData}.
+ * The deserialization schema knows Debezium's schema definition and can
extract the database data
+ * and convert into {@link RowData} with {@link RowKind}.
+ *
+ * Deserializes a byte[] message as a JSON object and reads
+ * the specified fields.
+ *
+ * Failures during deserialization are forwarded as wrapped IOExceptions.
+ *
+ * @see https://debezium.io/;>Debezium
+ */
+@Internal
+public final class DebeziumAvroDeserializationSchema implements
DeserializationSchema {
+ private static final long serialVersionUID = 1L;
+
+ /**
+* snapshot read
+*/
+ private static final String OP_READ = "r";
+ /**
+* insert operation
+*/
+ private static final String OP_CREATE = "c";
+ /**
+* update operation
+*/
+ private static final String OP_UPDATE = "u";
+ /**
+* delete operation
+*/
+ private static final String OP_DELETE = "d";
+
+ private static final String REPLICA_IDENTITY_EXCEPTION = "The
\"before\" field of %s message is null, " +
+ "if you are using Debezium Postgres Connector, " +
+ "please check the Postgres table has been set REPLICA IDENTITY
to FULL level.";
+
+ /**
+* The deserializer to deserialize Debezium Avro data.
+*/
+ private final AvroRowDataDeserializationSchema avroDeserializer;
+
+ /**
+* TypeInformation of the produced {@link RowData}.
+**/
+ private final TypeInformation resultTypeInfo;
+
+ /**
+* Debezium Avro data rowType
+*/
+ private final RowType rowType;
+
+
+ public DebeziumAvroDeserializationSchema(
+ RowType rowType,
+ TypeInformation resultTypeInfo,
+ String schemaRegistryUrl) {
+ this.resultTypeInfo = resultTypeInfo;
+ this.rowType = rowType;
+ RowType debeziumAvroRowType =
createDebeziumAvroRowType(fromLogicalToDataType(rowType));
+
+ this.avroDeserializer = new AvroRowDataDeserializationSchema(
+ ConfluentRegistryAvroDeserializationSchema.forGeneric(
+
AvroSchemaConverter.convertToSchema(debeziumAvroRowType),
+ schemaRegistryUrl),
+
AvroToRowDataConverters.createRowConverter(debeziumAvroRowType),
+ resultTypeInfo);
+ }
+
+ public DebeziumAvroDeserializationSchema(
+ RowType rowType,
+ TypeInformation resultTypeInfo,
+
[GitHub] [flink] shuiqiangchen commented on pull request #13322: [FLINK-17480][kubernetes] Support running PyFlink on Kubernetes.
shuiqiangchen commented on pull request #13322: URL: https://github.com/apache/flink/pull/13322#issuecomment-700386763 @wangyang0918 Thank you for your long time's help for reviewing this pr. @dianfu Could you please have a look at this PR since it has been approved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * ad9145cd549c242e40862c8ce21cf52b06927f00 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7003) * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * 921b49a2ccb91a91ff0f3c3cf176fec0966459ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7044) * c1ead553fe6892d93e5fc702f51d3aff397d7b78 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
KarmaGYZ commented on pull request #13473: URL: https://github.com/apache/flink/pull/13473#issuecomment-700384319 > Can we also add a test that tries to run 2 identical jobs at the same time? Thanks for the comment @zentol . I think it will be good to add a test for it. However, I'm not quite familiar with that logic. What I can find is the `ExecutionGraphConstructionTest`, we could check the `ExecutionAttemptID` for each execution vertex it generates. WDYT? Otherwise, I think we need to add an e2e test. Or do you have any suggestion? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
dianfu commented on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-700383952 @HuangXingBo Updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #13322: [FLINK-17480][kubernetes] Support running PyFlink on Kubernetes.
wangyang0918 commented on pull request #13322: URL: https://github.com/apache/flink/pull/13322#issuecomment-700382411 +1 for merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
HuangXingBo commented on a change in pull request #13492: URL: https://github.com/apache/flink/pull/13492#discussion_r496324046 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonOverAggregate.scala ## @@ -28,12 +29,10 @@ import org.apache.flink.table.functions.python.PythonFunctionInfo import org.apache.flink.table.planner.calcite.FlinkTypeFactory import org.apache.flink.table.planner.delegation.BatchPlanner import org.apache.flink.table.planner.plan.nodes.common.CommonPythonAggregate -import org.apache.flink.table.planner.plan.nodes.exec.ExecNode Review comment: unnecessary change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LakeShen commented on pull request #12781: [FLINK-18376][Table SQL/Runtime]Fix java.lang.ArrayIndexOutOfBoundsException in RetractableTopNFunction
LakeShen commented on pull request #12781: URL: https://github.com/apache/flink/pull/12781#issuecomment-700376927 Anyone to look this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
HuangXingBo commented on a change in pull request #13492:
URL: https://github.com/apache/flink/pull/13492#discussion_r496323126
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonGroupAggregate.scala
##
@@ -36,14 +37,12 @@ import
org.apache.flink.table.planner.plan.rules.physical.batch.BatchExecJoinRul
import org.apache.flink.table.planner.plan.utils.{FlinkRelOptUtil,
RelExplainUtil}
import org.apache.flink.table.runtime.typeutils.InternalTypeInfo
import org.apache.flink.table.types.logical.RowType
-
Review comment:
unnecessary change
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonCalc.scala
##
@@ -19,11 +19,10 @@
package org.apache.flink.table.planner.plan.nodes.physical.batch
import org.apache.flink.api.dag.Transformation
+import org.apache.flink.core.memory.ManagedMemoryUseCase
import org.apache.flink.table.data.RowData
import org.apache.flink.table.planner.delegation.BatchPlanner
import org.apache.flink.table.planner.plan.nodes.common.CommonPythonCalc
-import org.apache.flink.table.planner.plan.nodes.exec.ExecNode
-
Review comment:
unnecessary change
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonGroupWindowAggregate.scala
##
@@ -41,7 +42,6 @@ import
org.apache.flink.table.planner.plan.nodes.exec.{BatchExecNode, ExecNode}
import
org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecPythonGroupWindowAggregate.ARROW_PYTHON_GROUP_WINDOW_AGGREGATE_FUNCTION_OPERATOR_NAME
import org.apache.flink.table.runtime.typeutils.InternalTypeInfo
import org.apache.flink.table.types.logical.RowType
-
Review comment:
unnecessary change
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonOverAggregate.scala
##
@@ -28,12 +29,10 @@ import
org.apache.flink.table.functions.python.PythonFunctionInfo
import org.apache.flink.table.planner.calcite.FlinkTypeFactory
import org.apache.flink.table.planner.delegation.BatchPlanner
import org.apache.flink.table.planner.plan.nodes.common.CommonPythonAggregate
-import org.apache.flink.table.planner.plan.nodes.exec.ExecNode
Review comment:
unnecessary change
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonGroupAggregate.scala
##
@@ -36,14 +37,12 @@ import
org.apache.flink.table.planner.plan.rules.physical.batch.BatchExecJoinRul
import org.apache.flink.table.planner.plan.utils.{FlinkRelOptUtil,
RelExplainUtil}
import org.apache.flink.table.runtime.typeutils.InternalTypeInfo
import org.apache.flink.table.types.logical.RowType
-
import org.apache.calcite.plan.{RelOptCluster, RelOptRule, RelTraitSet}
import org.apache.calcite.rel.RelDistribution.Type.{HASH_DISTRIBUTED,
SINGLETON}
import org.apache.calcite.rel.`type`.RelDataType
import org.apache.calcite.rel.core.AggregateCall
import org.apache.calcite.rel.{RelCollationTraitDef, RelCollations, RelNode,
RelWriter}
import org.apache.calcite.util.{ImmutableIntList, Util}
-
Review comment:
ditto
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonOverAggregate.scala
##
@@ -28,12 +29,10 @@ import
org.apache.flink.table.functions.python.PythonFunctionInfo
import org.apache.flink.table.planner.calcite.FlinkTypeFactory
import org.apache.flink.table.planner.delegation.BatchPlanner
import org.apache.flink.table.planner.plan.nodes.common.CommonPythonAggregate
-import org.apache.flink.table.planner.plan.nodes.exec.ExecNode
import
org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecPythonOverAggregate.ARROW_PYTHON_OVER_WINDOW_AGGREGATE_FUNCTION_OPERATOR_NAME
import
org.apache.flink.table.planner.plan.utils.OverAggregateUtil.getLongBoundary
import org.apache.flink.table.runtime.typeutils.InternalTypeInfo
import org.apache.flink.table.types.logical.RowType
-
Review comment:
ditto
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamExecPythonGroupWindowAggregate.scala
##
@@ -31,15 +32,13 @@ import
org.apache.flink.table.planner.delegation.StreamPlanner
import org.apache.flink.table.planner.expressions.{PlannerRowtimeAttribute,
PlannerWindowEnd, PlannerWindowStart}
import org.apache.flink.table.planner.plan.logical.{LogicalWindow,
SlidingGroupWindow, TumblingGroupWindow}
import org.apache.flink.table.planner.plan.nodes.common.CommonPythonAggregate
-import
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * b5ac11907cbbe41f2b13e2b738458a3315d61663 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6993) * 0f253d15c6a2ab987bf2c824efb810a26c176fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7045) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13434: [FLINK-19292][hive] HiveCatalog should support specifying Hadoop conf dir with configuration
flinkbot edited a comment on pull request #13434: URL: https://github.com/apache/flink/pull/13434#issuecomment-695893232 ## CI report: * 98dfbc96735cb5ae978a526b046c2ec3790b6b4b UNKNOWN * b5ac11907cbbe41f2b13e2b738458a3315d61663 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6993) * 0f253d15c6a2ab987bf2c824efb810a26c176fff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19326) Allow explicitly configuring time behaviour on CEP PatternStream
[
https://issues.apache.org/jira/browse/FLINK-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203566#comment-17203566
]
Nicholas Jiang commented on FLINK-19326:
[~aljoscha]OK, I got the point you mentioned. I would like to add a setting for
processing time and event time.
> Allow explicitly configuring time behaviour on CEP PatternStream
>
>
> Key: FLINK-19326
> URL: https://issues.apache.org/jira/browse/FLINK-19326
> Project: Flink
> Issue Type: Sub-task
> Components: Library / CEP
>Reporter: Aljoscha Krettek
>Assignee: Nicholas Jiang
>Priority: Major
>
> With the deprecation of
> {{StreamExecutionEnvironment.setStreamTimeCharacteristic()}} in FLINK-19319
> we need a way of explicitly configuring the time behaviour of CEP operations.
> Currently, all CEP operations use the characteristic to configure themselves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * ad9145cd549c242e40862c8ce21cf52b06927f00 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7003) * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * 921b49a2ccb91a91ff0f3c3cf176fec0966459ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7044) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19401) Job stuck in restart loop due to "Could not find registered job manager"
[
https://issues.apache.org/jira/browse/FLINK-19401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203298#comment-17203298
]
Steven Zhen Wu edited comment on FLINK-19401 at 9/29/20, 12:25 AM:
---
[~trohrmann] emailed you the JM logs with INFO level logging.
It is not easily reproducible, maybe it happens once every 1-2 weeks for a
high-parallelism (~1,500) and large state (TBs) job. If DEBUG is required, we
can try to enable the DEBUG level logging and wait for it to happen again.
was (Author: stevenz3wu):
[~trohrmann] emailed you the JM logs with INFO level logging.
It is not easily reproducible, maybe it happens once every 1-2 weeks for a
high-parallelism (~1,500) and large state (TBs) job. If DEBUG is required, we
can try to enable the DEBUG level logging and wait for it happen again.
> Job stuck in restart loop due to "Could not find registered job manager"
>
>
> Key: FLINK-19401
> URL: https://issues.apache.org/jira/browse/FLINK-19401
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.1
>Reporter: Steven Zhen Wu
>Priority: Major
>
> Flink job sometimes got into a restart loop for many hours and can't recover
> until redeployed. We had some issue with Kafka that initially caused the job
> to restart.
> Below is the first of the many exceptions for "ResourceManagerException:
> Could not find registered job manager" error.
> {code}
> 2020-09-19 00:03:31,614 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{171f1df017dab3a42c032abd07908b9b}] and profile ResourceP
> rofile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,615 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{cc7d136c4ce1f32285edd4928e3ab2e2}] and profile
> ResourceProfile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,615 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{024c8a48dafaf8f07c49dd4320d5cc94}] and profile
> ResourceProfile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,615 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{a591eda805b3081ad2767f5641d0db06}] and profile
> ResourceProfile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,620 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher-35973] - Source: k2-csevpc ->
> k2-csevpcRaw -> (vhsPlaybackEvents -> Flat Map, merchImpressionsClientLog ->
> Flat Map) (56/640) (1b0d3dd1f19890886ff373a3f08809e8) switched from SCHEDULED
> to FAILED.
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> No pooled slot available and request to ResourceManager for new slot failed
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:433)
> at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$forward$21(FutureUtils.java:1065)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.forward(FutureUtils.java:1063)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager.createRootSlot(SlotSharingManager.java:155)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.allocateMultiTaskSlot(SchedulerImpl.java:511)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.allocateSharedSlot(SchedulerImpl.java:311)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.internalAllocateSlot(SchedulerImpl.java:160)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * ad9145cd549c242e40862c8ce21cf52b06927f00 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7003) * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * 921b49a2ccb91a91ff0f3c3cf176fec0966459ff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong closed pull request #13387: [Hotfix][typos]Modify debug log to print the compiling code
wangxlong closed pull request #13387: URL: https://github.com/apache/flink/pull/13387 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong closed pull request #11124: [Hotfix]Add suffix L to WatermarkGeneratorCodeGenTest#testAscendingWatermark long type fields
wangxlong closed pull request #11124: URL: https://github.com/apache/flink/pull/11124 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong closed pull request #11016: [FLINK-15677][TABLE-COMMON]Use physical field names for the default implementation of Tablesource#explainSource
wangxlong closed pull request #11016: URL: https://github.com/apache/flink/pull/11016 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * ad9145cd549c242e40862c8ce21cf52b06927f00 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7003) * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
dianfu commented on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-700329112 @HuangXingBo Thanks for the review. Updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13502: [FLINK-19392] Add the execution.runtime-mode
flinkbot edited a comment on pull request #13502: URL: https://github.com/apache/flink/pull/13502#issuecomment-700213477 ## CI report: * 145edc64fe7676b60df3fd322d771fb01711aaf8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7041) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
flinkbot edited a comment on pull request #13458: URL: https://github.com/apache/flink/pull/13458#issuecomment-697041219 ## CI report: * 1b59c5c3c61d9ea667f3e722bb54788d1d687cec Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7040) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19300) Timer loss after restoring from savepoint
[ https://issues.apache.org/jira/browse/FLINK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203504#comment-17203504 ] Xiang Gao commented on FLINK-19300: --- [~klion26] Thanks for taking a look! I'm not sure about the criteria for "blocker" issues. This issue seems to only exist in heap based timers, but losing timer is a bit dangerous. Wonder what's our process on patching this kind of issue. May I submit a PR for this? > Timer loss after restoring from savepoint > - > > Key: FLINK-19300 > URL: https://issues.apache.org/jira/browse/FLINK-19300 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Reporter: Xiang Gao >Priority: Critical > > While using heap-based timers, we are seeing occasional timer loss after > restoring program from savepoint, especially when using a remote savepoint > storage (s3). > After some investigation, the issue seems to be related to [this line in > deserialization|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/io/PostVersionedIOReadableWritable.java#L65]. > When trying to check the VERSIONED_IDENTIFIER, the input stream may not > guarantee filling the byte array, causing timers to be dropped for the > affected key group. > Should keep reading until expected number of bytes are actually read or if > end of the stream has been reached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13502: [FLINK-19392] Add the execution.runtime-mode
flinkbot edited a comment on pull request #13502: URL: https://github.com/apache/flink/pull/13502#issuecomment-700213477 ## CI report: * 145edc64fe7676b60df3fd322d771fb01711aaf8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7041) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13351: [FLINK-18990][task] Read channel state sequentially
flinkbot edited a comment on pull request #13351: URL: https://github.com/apache/flink/pull/13351#issuecomment-688741492 ## CI report: * d5de65d33ae7216bee7ae8d55bf20839584c520f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7039) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13502: [FLINK-19392] Add the execution.runtime-mode
flinkbot commented on pull request #13502: URL: https://github.com/apache/flink/pull/13502#issuecomment-700213477 ## CI report: * 145edc64fe7676b60df3fd322d771fb01711aaf8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13502: [FLINK-19392] Add the execution.runtime-mode
flinkbot commented on pull request #13502: URL: https://github.com/apache/flink/pull/13502#issuecomment-700209876 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 145edc64fe7676b60df3fd322d771fb01711aaf8 (Mon Sep 28 18:36:20 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kl0u opened a new pull request #13502: [FLINK-19392] Add the execution.runtime-mode
kl0u opened a new pull request #13502: URL: https://github.com/apache/flink/pull/13502 ## What is the purpose of the change This PR adds the `execution.runtime-mode`, as described in FLIP-134, without exposing it yet to the user. For now, we set the mode globally at the `StreamGraphGenerator`. An alternative would be to set it at each `Transformation` individually based on its predecessors. We went with the global option because, at least for now, we will schedule the whole graph either in STREAMING or in BATCH mode so the setting has to be global. In the future, if we consider mixed graphs where subgraphs are scheduled in BATCH mode and others in STREAMING, then we could consider setting the mode on each transformation individually. In addition, this PR sets the `Boundedness` of some Legacy sources to BOUNDED (e.g. `env.fromCollection()`). If this were to be combined with runtime.execution-mode set to AUTOMATIC, it would change the already exposed semantics. Given that we do not want that, we have set the default runtime.execution-mode to STREAMING so that the Boundedness is ignored and the exposed semantics are the same as before. ## Brief change log The main additions are: 1. the logic in the `StreamGraphGenerator.determineExecutionMode()`, which determines the `execution.runtime-mode` based on either an explicit setting, or by looking at the `Boundedness` of the sources of the pipeline and 2. the setting of Boundedness for some of the source in the `StreamExecutionEnvironment`. ## Verifying this change Add the `StreamGraphGeneratorRuntimeExecutionModeDetection` class with the relevant tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19392) Detect the execution mode based on the sources in the job.
[
https://issues.apache.org/jira/browse/FLINK-19392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19392:
---
Labels: pull-request-available (was: )
> Detect the execution mode based on the sources in the job.
> --
>
> Key: FLINK-19392
> URL: https://issues.apache.org/jira/browse/FLINK-19392
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
>
> As part of FLIP-134, we introduce the option {{execution.runtime-mode}} which
> can take the values: BATCH, STREAMING, and AUTOMATIC.
> In case of the latter, the system will scan the sources and detect if the job
> is to be execute either using batch scheduling or streaming. If all the
> sources are bounded, the system will go with BATCH, if at least one is
> unbounded, then the system will go with STREAMING.
> This issue targets introducing the logic of detecting the runtime mode based
> on the sources without exposing it yet to the user. The latter will happen in
> a follow-up issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
flinkbot edited a comment on pull request #13458: URL: https://github.com/apache/flink/pull/13458#issuecomment-697041219 ## CI report: * 3340d3b1f62c8cfe9e9b30fee877a4728d4b0837 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6978) * 1b59c5c3c61d9ea667f3e722bb54788d1d687cec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7040) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13438: [FLINK-19014][e2e] Increase startup timeout
flinkbot edited a comment on pull request #13438: URL: https://github.com/apache/flink/pull/13438#issuecomment-696045369 ## CI report: * 4e3397e2476210d441ceefab6802d4bbab1053c8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7037) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
flinkbot edited a comment on pull request #13458: URL: https://github.com/apache/flink/pull/13458#issuecomment-697041219 ## CI report: * 3340d3b1f62c8cfe9e9b30fee877a4728d4b0837 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6978) * 1b59c5c3c61d9ea667f3e722bb54788d1d687cec UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gm7y8 commented on pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
gm7y8 commented on pull request #13458: URL: https://github.com/apache/flink/pull/13458#issuecomment-700174911 @XComp I have taken care of commit message and other formatting issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-15578) Implement exactly-once JDBC sink
[ https://issues.apache.org/jira/browse/FLINK-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203347#comment-17203347 ] Kenzyme Le edited comment on FLINK-15578 at 9/28/20, 4:36 PM: -- Hi [~roman_khachatryan] , In case I am using a DB that doesn't support native 2PC, what would be an alternative solution you would recommend using JDBC? Thanks! was (Author: klden): Hi [~roman_khachatryan] , In case I am using a DB that doesn't support native 2PC, what would be an alternative solution using JDBC? Thanks! > Implement exactly-once JDBC sink > > > Key: FLINK-15578 > URL: https://issues.apache.org/jira/browse/FLINK-15578 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > > As per discussion in the dev mailing list, there are two options: > # Write-ahead log > # Two-phase commit (XA) > the latter being preferable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15578) Implement exactly-once JDBC sink
[ https://issues.apache.org/jira/browse/FLINK-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203347#comment-17203347 ] Kenzyme Le commented on FLINK-15578: Hi [~roman_khachatryan] , In case I am using a DB that doesn't support native 2PC, what would be an alternative solution using JDBC? Thanks! > Implement exactly-once JDBC sink > > > Key: FLINK-15578 > URL: https://issues.apache.org/jira/browse/FLINK-15578 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > > As per discussion in the dev mailing list, there are two options: > # Write-ahead log > # Two-phase commit (XA) > the latter being preferable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
flinkbot edited a comment on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-667140037 ## CI report: * 31abcb25fdf72394dbfa91d0ba9e151968375ad7 UNKNOWN * db99952f8166cb4d6337dcb3815e72ff2b49604b UNKNOWN * 33af84f70ab92a76818f1a4392fecb0c4a8e62f9 UNKNOWN * 12d5698e9185b3b97e5323485ee72617c4e535f9 UNKNOWN * b7162b6109b03f905f2c52cc22002b198bc9438e UNKNOWN * aaf4aca0d5ce666001035f6ae66bca841f87c74d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7036) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #13499: [FLINK-16972][network] LocalBufferPool eagerly fetches global segments to ensure proper availability.
pnowojski commented on a change in pull request #13499:
URL: https://github.com/apache/flink/pull/13499#discussion_r495970800
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java
##
@@ -48,8 +48,6 @@
* minimum number of network buffers in this pool
* @param maxUsedBuffers
* maximum number of network buffers this pool offers
-* @param bufferPoolOwner
Review comment:
Have you checked if the `BufferPoolOwner` is not part of our shuffle
service API? Maybe there can be some 3rd party shuffle services using it?
@zhijiangW seemed to be fine with removing it in the ticket, so I guess
that's not an issue (he was involved in the plugable shuffle service story).
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -518,6 +518,11 @@ private void readRecoveredChannelState() throws
IOException, InterruptedExceptio
"Cannot restore state
to a non-checkpointable partition type: " + writer);
}
}
+
+ if (!recordWriter.isAvailable()) {
+ MailboxDefaultAction.Suspension
suspendedDefaultAction = mailboxProcessor.suspendDefaultAction();
+
getInputOutputJointFuture(InputStatus.NOTHING_AVAILABLE).thenRun(suspendedDefaultAction::resume);
+ }
Review comment:
What's the purpose of this change? Is it actually working? What if
`getInputOutputJointFuture` returns completed future, but it become unavailable
during the input recovery?
Also, it's missing a test coverage.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
##
@@ -322,23 +350,71 @@ private MemorySegment requestMemorySegment() {
return requestMemorySegment(UNKNOWN_CHANNEL);
}
- @Nullable
- private MemorySegment requestMemorySegmentFromGlobal() {
- assert Thread.holdsLock(availableMemorySegments);
+ private boolean requestMemorySegmentFromGlobal() {
+ if (numberOfRequestedMemorySegments >= currentPoolSize) {
+ return false;
+ }
+
+ MemorySegment segment =
networkBufferPool.requestMemorySegment();
+ if (segment != null) {
+ availableMemorySegments.add(segment);
+ numberOfRequestedMemorySegments++;
+ return true;
+ }
+ return false;
+ }
- if (isDestroyed) {
- throw new IllegalStateException("Buffer pool is
destroyed.");
+ /**
+* Tries to obtain a buffer from global pool as soon as one pool is
available. Note that multiple
+* {@link LocalBufferPool}s might wait on the future of the global
pool, hence this method double-check if a new
+* buffer is really needed at the time it becomes available.
+*/
+ private void eagerlyRequestMemorySegmentFromGlobal() {
+ if (eagerlyRequesting) {
+ return;
}
+ eagerlyRequesting = true;
+ networkBufferPool.getAvailableFuture().thenRun(() -> {
+ eagerlyRequesting = false;
+ if (availabilityHelper.isAvailable()) {
+ // there is currently no benefit for this pool
to obtain buffer from global; give other pools precedent
+ return;
+ }
Review comment:
doing this under the lock would look more consistent? As it is, can not
it yield false results? All of the modifications to the `availabilityHelper `
are happening under the `availableMemorySegments` lock , so after moving there
it should be perfectly accurate.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
##
@@ -322,23 +350,71 @@ private MemorySegment requestMemorySegment() {
return requestMemorySegment(UNKNOWN_CHANNEL);
}
- @Nullable
- private MemorySegment requestMemorySegmentFromGlobal() {
- assert Thread.holdsLock(availableMemorySegments);
+ private boolean requestMemorySegmentFromGlobal() {
+ if (numberOfRequestedMemorySegments >= currentPoolSize) {
+ return false;
+ }
+
+ MemorySegment segment =
networkBufferPool.requestMemorySegment();
+ if (segment != null) {
+ availableMemorySegments.add(segment);
+ numberOfRequestedMemorySegments++;
+ return true;
[GitHub] [flink] zentol commented on pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
zentol commented on pull request #13473: URL: https://github.com/apache/flink/pull/13473#issuecomment-700122923 Can we also add a test that tries to run 2 identical jobs at the same time? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13500: [FLINK-19180][state backends] Make RocksDB respect managed memory fraction
flinkbot edited a comment on pull request #13500: URL: https://github.com/apache/flink/pull/13500#issuecomment-699908639 ## CI report: * bee5be24b69942ac4b6323da5bd701e8287f145b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7035) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13482: test
flinkbot edited a comment on pull request #13482: URL: https://github.com/apache/flink/pull/13482#issuecomment-698903502 ## CI report: * b6d1f311cb2a041680b89e3aedfe3d5cbdca42d0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7034) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13216: [FLINK-18999][table-planner-blink][hive] Temporary generic table does…
flinkbot edited a comment on pull request #13216: URL: https://github.com/apache/flink/pull/13216#issuecomment-678268420 ## CI report: * 528277a79d67bb9309ec3d6121d40dfca9e83553 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7033) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19401) Job stuck in restart loop due to "Could not find registered job manager"
[
https://issues.apache.org/jira/browse/FLINK-19401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203298#comment-17203298
]
Steven Zhen Wu commented on FLINK-19401:
[~trohrmann] emailed you the JM logs with INFO level logging.
It is not easily reproducible, maybe it happens once every 1-2 weeks for a
high-parallelism (~1,500) and large state (TBs) job. If DEBUG is required, we
can try to enable the DEBUG level logging and wait for it happen again.
> Job stuck in restart loop due to "Could not find registered job manager"
>
>
> Key: FLINK-19401
> URL: https://issues.apache.org/jira/browse/FLINK-19401
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.1
>Reporter: Steven Zhen Wu
>Priority: Major
>
> Flink job sometimes got into a restart loop for many hours and can't recover
> until redeployed. We had some issue with Kafka that initially caused the job
> to restart.
> Below is the first of the many exceptions for "ResourceManagerException:
> Could not find registered job manager" error.
> {code}
> 2020-09-19 00:03:31,614 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{171f1df017dab3a42c032abd07908b9b}] and profile ResourceP
> rofile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,615 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{cc7d136c4ce1f32285edd4928e3ab2e2}] and profile
> ResourceProfile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,615 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{024c8a48dafaf8f07c49dd4320d5cc94}] and profile
> ResourceProfile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,615 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl
> [flink-akka.actor.default-dispatcher-35973] - Requesting new slot
> [SlotRequestId{a591eda805b3081ad2767f5641d0db06}] and profile
> ResourceProfile{UNKNOWN} from resource manager.
> 2020-09-19 00:03:31,620 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher-35973] - Source: k2-csevpc ->
> k2-csevpcRaw -> (vhsPlaybackEvents -> Flat Map, merchImpressionsClientLog ->
> Flat Map) (56/640) (1b0d3dd1f19890886ff373a3f08809e8) switched from SCHEDULED
> to FAILED.
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> No pooled slot available and request to ResourceManager for new slot failed
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:433)
> at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$forward$21(FutureUtils.java:1065)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.forward(FutureUtils.java:1063)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager.createRootSlot(SlotSharingManager.java:155)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.allocateMultiTaskSlot(SchedulerImpl.java:511)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.allocateSharedSlot(SchedulerImpl.java:311)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.internalAllocateSlot(SchedulerImpl.java:160)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.allocateSlotInternal(SchedulerImpl.java:143)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.allocateSlot(SchedulerImpl.java:113)
> at
> org.apache.flink.runtime.executiongraph.SlotProviderStrategy$NormalSlotProviderStrategy.allocateSlot(SlotProviderStrategy.java:115)
> at
>
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13322: [FLINK-17480][kubernetes] Support running PyFlink on Kubernetes.
shuiqiangchen commented on a change in pull request #13322:
URL: https://github.com/apache/flink/pull/13322#discussion_r496041819
##
File path:
flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh
##
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common_kubernetes.sh
+
+CURRENT_DIR=`cd "$(dirname "$0")" && pwd -P`
+CLUSTER_ROLE_BINDING="flink-role-binding-default"
+CLUSTER_ID="flink-native-k8s-pyflink-application-1"
+PURE_FLINK_IMAGE_NAME="test_kubernetes_application"
+PYFLINK_IMAGE_NAME="test_kubernetes_pyflink_application"
+LOCAL_LOGS_PATH="${TEST_DATA_DIR}/log"
+
+function internal_cleanup {
+kubectl delete deployment ${CLUSTER_ID}
+kubectl delete clusterrolebinding ${CLUSTER_ROLE_BINDING}
+}
+
+start_kubernetes
+
+build_image ${PURE_FLINK_IMAGE_NAME}
+
+# Build PyFlink wheel package
Review comment:
Yes, the e2e test will first build a flink base image using the local
flink binary, and then it will turn to the local `flink-python` module to build
a pyflink dist which will be added and installed into the new dedicated pyflink
image. And then run the e2e test with the new image. The overall elapsed time
is about 10 minutes, most of time is for building the image.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19154) Application mode deletes HA data in case of suspended ZooKeeper connection
[
https://issues.apache.org/jira/browse/FLINK-19154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203288#comment-17203288
]
Kostas Kloudas commented on FLINK-19154:
I will have a look.
> Application mode deletes HA data in case of suspended ZooKeeper connection
> --
>
> Key: FLINK-19154
> URL: https://issues.apache.org/jira/browse/FLINK-19154
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.12.0, 1.11.1
> Environment: Run a stand-alone cluster that runs a single job (if you
> are familiar with the way Ververica Platform runs Flink jobs, we use a very
> similar approach). It runs Flink 1.11.1 straight from the official docker
> image.
>Reporter: Husky Zeng
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> A user reported that Flink's application mode deletes HA data in case of a
> suspended ZooKeeper connection [1].
> The problem seems to be that the {{ApplicationDispatcherBootstrap}} class
> produces an exception (that the request job can no longer be found because of
> a lost ZooKeeper connection) which will be interpreted as a job failure. Due
> to this interpretation, the cluster will be shut down with a terminal state
> of FAILED which will cause the HA data to be cleaned up. The exact problem
> occurs in the {{JobStatusPollingUtils.getJobResult}} which is called by
> {{ApplicationDispatcherBootstrap.getJobResult()}}.
> The above described behaviour can be found in this log [2].
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-metadata-deleted-by-Flink-after-ZK-connection-issues-td37937.html
> [2] https://pastebin.com/raw/uH9KDU2L
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13351: [FLINK-18990][task] Read channel state sequentially
flinkbot edited a comment on pull request #13351: URL: https://github.com/apache/flink/pull/13351#issuecomment-688741492 ## CI report: * 859b19e0af50e8188addfa7ce42179b25c806c80 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6972) * d5de65d33ae7216bee7ae8d55bf20839584c520f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7039) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13316: [FLINK-14422][runtime] Expose network memory usage to TaskManagerDetailsHandler's endpoint
tillrohrmann commented on a change in pull request #13316:
URL: https://github.com/apache/flink/pull/13316#discussion_r495997147
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java
##
@@ -271,12 +271,28 @@ public int getTotalNumberOfMemorySegments() {
return totalNumberOfMemorySegments;
}
+ public long getTotalMemory() {
+ return getTotalNumberOfMemorySegments() * memorySegmentSize;
+ }
+
public int getNumberOfAvailableMemorySegments() {
synchronized (availableMemorySegments) {
return availableMemorySegments.size();
}
}
+ public long getAvailableMemory() {
+ return getNumberOfAvailableMemorySegments() * memorySegmentSize;
+ }
+
+ public int getNumberOfUsedMemorySegments() {
+ return getTotalNumberOfMemorySegments() -
getNumberOfAvailableMemorySegments();
+ }
+
+ public long getUsedMemory() {
+ return getNumberOfUsedMemorySegments() * memorySegmentSize;
+ }
Review comment:
I see your point. However, `LocalBufferPools` can contain excess memory
which can be claimed by other `LocalBufferPools`. See
`LocalBufferPool.returnExcessMemorySegments` for more details. Hence it is not
correct that memory which is assigned to a `LocalBufferPool` won't be available
for other buffer pools.
Ideally the used network memory metric would allow us to see if we have
overprovisioned a TM with network memory or whether it it running at its
capacity. However, this won't be visible if we report how many memory segments
have been handed out by the `NetworkBufferPool` to the `LocalBufferPools` where
they might lie around idling.
If there is no easy way to achieve this goal, then I would be ok with
keeping as it is which gives a more coarse grained overview over the resource
utilization which is already a plus.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-18795) Support for HBase 2
[ https://issues.apache.org/jira/browse/FLINK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-18795. -- Resolution: Fixed > Support for HBase 2 > --- > > Key: FLINK-18795 > URL: https://issues.apache.org/jira/browse/FLINK-18795 > Project: Flink > Issue Type: New Feature > Components: Connectors / HBase >Affects Versions: 1.11.1 >Reporter: Felipe Lolas >Assignee: Miklos Gergely >Priority: Minor > Labels: pull-request-available > > Hi! > Would be nice to have support for HBase 2. I have a working solution, but I > need some advice for the PR. > * Should we have a separate connector for HBase2 like in Kafka? > * flink-yarn is affected because API for getting a Token changed(no breaking > changes). having said that, is it's okay to add that change in the PR in > another commit? > Thanks, > Felipe -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18795) Support for HBase 2
[ https://issues.apache.org/jira/browse/FLINK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203272#comment-17203272 ] Gyula Fora commented on FLINK-18795: master: 16c97c8db718219ed036218faf3519541f24b430 > Support for HBase 2 > --- > > Key: FLINK-18795 > URL: https://issues.apache.org/jira/browse/FLINK-18795 > Project: Flink > Issue Type: New Feature > Components: Connectors / HBase >Affects Versions: 1.11.1 >Reporter: Felipe Lolas >Assignee: Miklos Gergely >Priority: Minor > Labels: pull-request-available > > Hi! > Would be nice to have support for HBase 2. I have a working solution, but I > need some advice for the PR. > * Should we have a separate connector for HBase2 like in Kafka? > * flink-yarn is affected because API for getting a Token changed(no breaking > changes). having said that, is it's okay to add that change in the PR in > another commit? > Thanks, > Felipe -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-9849) Create hbase connector for hbase version to 2.0.2
[ https://issues.apache.org/jira/browse/FLINK-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-9849. - Resolution: Fixed > Create hbase connector for hbase version to 2.0.2 > - > > Key: FLINK-9849 > URL: https://issues.apache.org/jira/browse/FLINK-9849 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase >Reporter: Ted Yu >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > Attachments: hbase-2.1.0.dep > > > Currently hbase 1.4.3 is used for hbase connector. > We should create connector for hbase 2.0.2 which would be released. > Since there are API changes for the 2.0.2 release, a new hbase connector is > desirable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19237) LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with "NoResourceAvailableException: Could not allocate the required slot within slot request timeout
[
https://issues.apache.org/jira/browse/FLINK-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203268#comment-17203268
]
Robert Metzger commented on FLINK-19237:
Since I've worked on FLINK-16866, I'll take a look at the issue.
> LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with
> "NoResourceAvailableException: Could not allocate the required slot within
> slot request timeout"
>
>
> Key: FLINK-19237
> URL: https://issues.apache.org/jira/browse/FLINK-19237
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Matthias
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6499=logs=6bfdaf55-0c08-5e3f-a2d2-2a0285fd41cf=fd9796c3-9ce8-5619-781c-42f873e126a6]
> {code}
> 2020-09-14T21:11:02.8200203Z [ERROR]
> testReelectionOfJobMaster(org.apache.flink.runtime.leaderelection.LeaderChangeClusterComponentsTest)
> Time elapsed: 300.14 s <<< FAILURE!
> 2020-09-14T21:11:02.8201761Z java.lang.AssertionError: Job failed.
> 2020-09-14T21:11:02.8202749Z at
> org.apache.flink.runtime.jobmaster.utils.JobResultUtils.throwAssertionErrorOnFailedResult(JobResultUtils.java:54)
> 2020-09-14T21:11:02.8203794Z at
> org.apache.flink.runtime.jobmaster.utils.JobResultUtils.assertSuccess(JobResultUtils.java:30)
> 2020-09-14T21:11:02.8205177Z at
> org.apache.flink.runtime.leaderelection.LeaderChangeClusterComponentsTest.testReelectionOfJobMaster(LeaderChangeClusterComponentsTest.java:152)
> 2020-09-14T21:11:02.8206191Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-09-14T21:11:02.8206985Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-09-14T21:11:02.8207930Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-09-14T21:11:02.8208927Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-09-14T21:11:02.8209753Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-09-14T21:11:02.8210710Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-09-14T21:11:02.8211608Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-09-14T21:11:02.8214473Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-09-14T21:11:02.8215398Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-09-14T21:11:02.8216199Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-09-14T21:11:02.8216947Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-09-14T21:11:02.8217695Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-09-14T21:11:02.8218635Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-09-14T21:11:02.8219499Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-09-14T21:11:02.8220313Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-09-14T21:11:02.8221060Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-09-14T21:11:02.8222171Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-09-14T21:11:02.8222937Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-09-14T21:11:02.8223688Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-09-14T21:11:02.8225191Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-09-14T21:11:02.8226086Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-09-14T21:11:02.8226761Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-09-14T21:11:02.8227453Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-09-14T21:11:02.8228392Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-09-14T21:11:02.8229256Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-09-14T21:11:02.8235798Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-09-14T21:11:02.8237650Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>
[GitHub] [flink] flinkbot edited a comment on pull request #13499: [FLINK-16972][network] LocalBufferPool eagerly fetches global segments to ensure proper availability.
flinkbot edited a comment on pull request #13499: URL: https://github.com/apache/flink/pull/13499#issuecomment-699852497 ## CI report: * 378555e68fc64c708f033109422f8701013fdaed Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7031) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13351: [FLINK-18990][task] Read channel state sequentially
flinkbot edited a comment on pull request #13351: URL: https://github.com/apache/flink/pull/13351#issuecomment-688741492 ## CI report: * 859b19e0af50e8188addfa7ce42179b25c806c80 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6972) * d5de65d33ae7216bee7ae8d55bf20839584c520f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] miklosgergely commented on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
miklosgergely commented on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-700049806 Thank you @flolas @gyfora @leonardBang @mbalassi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17979) Support rescaling for Unaligned Checkpoints
[ https://issues.apache.org/jira/browse/FLINK-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-17979: Parent: FLINK-19442 Issue Type: Sub-task (was: Improvement) > Support rescaling for Unaligned Checkpoints > --- > > Key: FLINK-17979 > URL: https://issues.apache.org/jira/browse/FLINK-17979 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Roman Khachatryan >Priority: Major > Fix For: 1.12.0 > > > This is one of the limitations of Unaligned Checkpoints MVP. > (see [Unaligned checkpoints: recovery & > rescaling|https://docs.google.com/document/d/1T2WB163uf8xt6Eu2JS0Jyy2XZyF4YpnzGiHlo6twrks/edit?usp=sharing] > for possible options) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] asfgit closed pull request #13128: [FLINK-18795][hbase] Support for HBase 2
asfgit closed pull request #13128: URL: https://github.com/apache/flink/pull/13128 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19442) Unaligned checkpoints 1.1
[ https://issues.apache.org/jira/browse/FLINK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-19442: Component/s: Runtime / Checkpointing > Unaligned checkpoints 1.1 > - > > Key: FLINK-19442 > URL: https://issues.apache.org/jira/browse/FLINK-19442 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Reporter: Arvid Heise >Priority: Major > > This is an umbrella issue for all efforts that increase the usability of > Unaligned checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19442) Unaligned checkpoints 1.1
[ https://issues.apache.org/jira/browse/FLINK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-19442: Issue Type: Improvement (was: Bug) > Unaligned checkpoints 1.1 > - > > Key: FLINK-19442 > URL: https://issues.apache.org/jira/browse/FLINK-19442 > Project: Flink > Issue Type: Improvement >Reporter: Arvid Heise >Priority: Major > > This is an umbrella issue for all efforts that increase the usability of > Unaligned checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19442) Unaligned checkpoints 1.1
Arvid Heise created FLINK-19442: --- Summary: Unaligned checkpoints 1.1 Key: FLINK-19442 URL: https://issues.apache.org/jira/browse/FLINK-19442 Project: Flink Issue Type: Bug Reporter: Arvid Heise This is an umbrella issue for all efforts that increase the usability of Unaligned checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18404) Make unaligned checkpoints rescalable
[ https://issues.apache.org/jira/browse/FLINK-18404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise closed FLINK-18404. --- Resolution: Duplicate > Make unaligned checkpoints rescalable > - > > Key: FLINK-18404 > URL: https://issues.apache.org/jira/browse/FLINK-18404 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.12.0 >Reporter: Arvid Heise >Priority: Critical > > Currently, the in-flight data of unaligned checkpoints is not handled > properly in rescaling scenarios. We guarded with a fail-safe exception > against it. A proper rescaling support would increase the benefits of > unaligned checkpoints. > To rescale in-flight data, the state shard needs to be redistributed across > all new assigned tasks and some additional filter needs to ensure that only > the relevant key-groups are actually recovered (or processed). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol commented on a change in pull request #13356: [FLINK-16789][runtime][rest] Enable JMX RMI port retrieval via REST API
zentol commented on a change in pull request #13356: URL: https://github.com/apache/flink/pull/13356#discussion_r495976247 ## File path: docs/_includes/generated/jmx_server_configuration.html ## @@ -1,18 +0,0 @@ - - - -Key Review comment: this shouldn't be deleted; even if options are included in specific sections we also always generate a separate table for each originating file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13438: [FLINK-19014][e2e] Increase startup timeout
flinkbot edited a comment on pull request #13438: URL: https://github.com/apache/flink/pull/13438#issuecomment-696045369 ## CI report: * 18f0204661403cc715f0daccc44b52965593ad69 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6696) * 4e3397e2476210d441ceefab6802d4bbab1053c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7037) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 313b80f4474455d7013b0852929b5a8458f391a1 UNKNOWN * 8dcdccd0b97ed873d628f831642786103b8f8878 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7028) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7024) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
