[jira] [Commented] (FLINK-31133) PartiallyFinishedSourcesITCase hangs if a checkpoint fails
[
https://issues.apache.org/jira/browse/FLINK-31133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695540#comment-17695540
]
Yu Li commented on FLINK-31133:
---
Thanks for the quick action [~roman]
> PartiallyFinishedSourcesITCase hangs if a checkpoint fails
> --
>
> Key: FLINK-31133
> URL: https://issues.apache.org/jira/browse/FLINK-31133
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.3, 1.16.1, 1.18.0, 1.17.1
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.16.2, 1.18.0, 1.17.1, 1.15.5
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b
> This build ran into a timeout. Based on the stacktraces reported, it was
> either caused by
> [SnapshotMigrationTestBase.restoreAndExecute|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=13475]:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f23d800b800 nid=0x60cdd waiting on
> condition [0x7f23e1c0d000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.test.checkpointing.utils.SnapshotMigrationTestBase.restoreAndExecute(SnapshotMigrationTestBase.java:382)
> at
> org.apache.flink.test.migration.TypeSerializerSnapshotMigrationITCase.testSnapshot(TypeSerializerSnapshotMigrationITCase.java:172)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> [...]
> {code}
> or
> [PartiallyFinishedSourcesITCase.test|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=10401]:
> {code}
> 2023-02-20T07:13:05.6084711Z "main" #1 prio=5 os_prio=0
> tid=0x7fd35c00b800 nid=0x8c8a waiting on condition [0x7fd363d0f000]
> 2023-02-20T07:13:05.6085149Zjava.lang.Thread.State: TIMED_WAITING
> (sleeping)
> 2023-02-20T07:13:05.6085487Z at java.lang.Thread.sleep(Native Method)
> 2023-02-20T07:13:05.6085925Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
> 2023-02-20T07:13:05.6086512Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:138)
> 2023-02-20T07:13:05.6087103Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForSubtasksToFinish(CommonTestUtils.java:291)
> 2023-02-20T07:13:05.6087730Z at
> org.apache.flink.runtime.operators.lifecycle.TestJobExecutor.waitForSubtasksToFinish(TestJobExecutor.java:226)
> 2023-02-20T07:13:05.6088410Z at
> org.apache.flink.runtime.operators.lifecycle.PartiallyFinishedSourcesITCase.test(PartiallyFinishedSourcesITCase.java:138)
> 2023-02-20T07:13:05.6088957Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
> Still, it sounds odd: Based on a code analysis it's quite unlikely that those
> two caused the issue. The former one has a 5 min timeout (see related code in
> [SnapshotMigrationTestBase:382|https://github.com/apache/flink/blob/release-1.15/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SnapshotMigrationTestBase.java#L382]).
> For the other one, we found it being not responsible in the past when some
> other concurrent test caused the issue (see FLINK-30261).
> An investigation on where we lose the time for the timeout revealed that
> {{AdaptiveSchedulerITCase}} took 2980s to finish (see [build
> logs|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=5265]).
> {code}
> 2023-02-20T03:43:55.4546050Z Feb 20 03:43:55 [ERROR] Picked up
> JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> 2023-02-20T03:43:58.0448506Z Feb 20 03:43:58 [INFO] Running
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
> 2023-02-20T04:33:38.6824634Z Feb 20 04:33:38 [INFO] Tests run: 6, Failures:
> 0, Errors: 0, Skipped: 0, Time elapsed: 2,980.445 s - in
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31133) PartiallyFinishedSourcesITCase hangs if a checkpoint fails
[
https://issues.apache.org/jira/browse/FLINK-31133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695501#comment-17695501
]
Yu Li commented on FLINK-31133:
---
The 1.15.4 version is about to release with [RC under
vote|https://lists.apache.org/thread/4463cypc257l7j9rj2pycofbsdbbjx59]. Please
check and confirm whether this issue could still make into 1.15.4 and move it
out if not. Thanks.
> PartiallyFinishedSourcesITCase hangs if a checkpoint fails
> --
>
> Key: FLINK-31133
> URL: https://issues.apache.org/jira/browse/FLINK-31133
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.3, 1.16.1, 1.18.0, 1.17.1
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.15.4, 1.16.2, 1.18.0, 1.17.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b
> This build ran into a timeout. Based on the stacktraces reported, it was
> either caused by
> [SnapshotMigrationTestBase.restoreAndExecute|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=13475]:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f23d800b800 nid=0x60cdd waiting on
> condition [0x7f23e1c0d000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.test.checkpointing.utils.SnapshotMigrationTestBase.restoreAndExecute(SnapshotMigrationTestBase.java:382)
> at
> org.apache.flink.test.migration.TypeSerializerSnapshotMigrationITCase.testSnapshot(TypeSerializerSnapshotMigrationITCase.java:172)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> [...]
> {code}
> or
> [PartiallyFinishedSourcesITCase.test|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=10401]:
> {code}
> 2023-02-20T07:13:05.6084711Z "main" #1 prio=5 os_prio=0
> tid=0x7fd35c00b800 nid=0x8c8a waiting on condition [0x7fd363d0f000]
> 2023-02-20T07:13:05.6085149Zjava.lang.Thread.State: TIMED_WAITING
> (sleeping)
> 2023-02-20T07:13:05.6085487Z at java.lang.Thread.sleep(Native Method)
> 2023-02-20T07:13:05.6085925Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
> 2023-02-20T07:13:05.6086512Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:138)
> 2023-02-20T07:13:05.6087103Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForSubtasksToFinish(CommonTestUtils.java:291)
> 2023-02-20T07:13:05.6087730Z at
> org.apache.flink.runtime.operators.lifecycle.TestJobExecutor.waitForSubtasksToFinish(TestJobExecutor.java:226)
> 2023-02-20T07:13:05.6088410Z at
> org.apache.flink.runtime.operators.lifecycle.PartiallyFinishedSourcesITCase.test(PartiallyFinishedSourcesITCase.java:138)
> 2023-02-20T07:13:05.6088957Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
> Still, it sounds odd: Based on a code analysis it's quite unlikely that those
> two caused the issue. The former one has a 5 min timeout (see related code in
> [SnapshotMigrationTestBase:382|https://github.com/apache/flink/blob/release-1.15/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SnapshotMigrationTestBase.java#L382]).
> For the other one, we found it being not responsible in the past when some
> other concurrent test caused the issue (see FLINK-30261).
> An investigation on where we lose the time for the timeout revealed that
> {{AdaptiveSchedulerITCase}} took 2980s to finish (see [build
> logs|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46299=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=5265]).
> {code}
> 2023-02-20T03:43:55.4546050Z Feb 20 03:43:55 [ERROR] Picked up
> JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> 2023-02-20T03:43:58.0448506Z Feb 20 03:43:58 [INFO] Running
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
> 2023-02-20T04:33:38.6824634Z Feb 20 04:33:38 [INFO] Tests run: 6, Failures:
> 0, Errors: 0, Skipped: 0, Time elapsed: 2,980.445 s - in
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-28910) CDC From Mysql To Hbase Bugs
[ https://issues.apache.org/jira/browse/FLINK-28910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601628#comment-17601628 ] Yu Li edited comment on FLINK-28910 at 9/8/22 4:47 AM: --- >From my point of view, a better solution is to facilitate the atomic >operations in HBase (Table#checkAndMutate or Table#mutateRow) for the >update-alike change data ingestion. However, currently Flink's >`HBaseSinkFunction` takes usage of HBase's `BufferedMutator` and >`BufferedMutator` only supports `Mutation` (actually >[currently|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java#L93-L99] > only Put and Delete), and operations like `RowMutations` or `CheckAndMuate` >are not supported. Let me check whether we could do something on the HBase >side. cc [~zhangduo] was (Author: carp84): >From my point of view, a better solution is to facilitate the atomic >operations in HBase (Table#checkAndMutate or Table#muateRow) for the >update-alike change data ingestion. However, currently Flink's >`HBaseSinkFunction` takes usage of HBase's `BufferedMutator` and >`BufferedMutator` only supports `Mutation` (actually >[currently|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java#L93-L99] > only Put and Delete), and operations like `RowMutations` or `CheckAndMuate` >are not supported. Let me check whether we could do something on the HBase >side. cc [~zhangduo] > CDC From Mysql To Hbase Bugs > > > Key: FLINK-28910 > URL: https://issues.apache.org/jira/browse/FLINK-28910 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Reporter: TE >Priority: Major > Labels: pull-request-available, stale-blocker > > I use Flink for CDC from Mysql to Hbase. > The problem I encountered is that the Mysql record is updated (not deleted), > but the record in hbase is deleted sometimes. > I tried to analyze the problem and found the reason as follows: > The update action of Mysql will be decomposed into delete + insert by Flink. > The Hbase connector uses a mutator to handle this set of actions. > However, if the order of these actions is not actively set, the processing of > the mutator will not guarantee the order of execution. > Therefore, when the update of Mysql is triggered, it is possible that hbase > actually performed the actions in the order of put + delete, resulting in the > data being deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28910) CDC From Mysql To Hbase Bugs
[ https://issues.apache.org/jira/browse/FLINK-28910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601628#comment-17601628 ] Yu Li commented on FLINK-28910: --- >From my point of view, a better solution is to facilitate the atomic >operations in HBase (Table#checkAndMutate or Table#muateRow) for the >update-alike change data ingestion. However, currently Flink's >`HBaseSinkFunction` takes usage of HBase's `BufferedMutator` and >`BufferedMutator` only supports `Mutation` (actually >[currently|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java#L93-L99] > only Put and Delete), and operations like `RowMutations` or `CheckAndMuate` >are not supported. Let me check whether we could do something on the HBase >side. cc [~zhangduo] > CDC From Mysql To Hbase Bugs > > > Key: FLINK-28910 > URL: https://issues.apache.org/jira/browse/FLINK-28910 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Reporter: TE >Priority: Major > Labels: pull-request-available, stale-blocker > > I use Flink for CDC from Mysql to Hbase. > The problem I encountered is that the Mysql record is updated (not deleted), > but the record in hbase is deleted sometimes. > I tried to analyze the problem and found the reason as follows: > The update action of Mysql will be decomposed into delete + insert by Flink. > The Hbase connector uses a mutator to handle this set of actions. > However, if the order of these actions is not actively set, the processing of > the mutator will not guarantee the order of execution. > Therefore, when the update of Mysql is triggered, it is possible that hbase > actually performed the actions in the order of put + delete, resulting in the > data being deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27655) Implement Avro File statistic collector
[ https://issues.apache.org/jira/browse/FLINK-27655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27655: -- Component/s: Table Store > Implement Avro File statistic collector > --- > > Key: FLINK-27655 > URL: https://issues.apache.org/jira/browse/FLINK-27655 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Zheng Hu >Priority: Minor > Fix For: table-store-0.2.0 > > > Currently, the flink table store's avro file writer don't provide its File > statistic collector. So we have to use the generic FieldStatsCollector. > In fact, the correct direction is: Making all format writer has their own > FileStatsCollector, so that we can just parse the columnar statistic from the > file tailer, instead of comparing each column max-min when writing the > records into the columnar file. > In this way, I think we can just remove the FileFormatImpl class and > FieldStatsCollector class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27517) Introduce rolling file writer to write one record each time for append-only table.
[ https://issues.apache.org/jira/browse/FLINK-27517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27517: -- Component/s: Table Store > Introduce rolling file writer to write one record each time for append-only > table. > -- > > Key: FLINK-27517 > URL: https://issues.apache.org/jira/browse/FLINK-27517 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > Currently, we table store has introduced a `RollingFile` to write an > iterator of rows into the underlying files. That's suitable for the memory > store flush processing, but for append-only table, it usually don't have any > memory store to cache those branch of rows temporarily, the idea approach is > writing one record into the columnar writer each time, and the columnar > writer will cache them and write into the underlying file in one batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27546) Add append only writer which implements the RecordWriter interface.
[ https://issues.apache.org/jira/browse/FLINK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27546: -- Component/s: Table Store > Add append only writer which implements the RecordWriter interface. > --- > > Key: FLINK-27546 > URL: https://issues.apache.org/jira/browse/FLINK-27546 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > We already has a DataFileWriter in our current flink table store, but this > DataFileWriter was designed to flush the records which were sorted by their > keys. > For the append-only table, the records to write will not have any keys or > sort orders. So let's introduce an append-only writer to ingest the > append-only records. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27580) Implement filter pushdown for TableStoreHiveStorageHandler
[
https://issues.apache.org/jira/browse/FLINK-27580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-27580:
--
Component/s: Table Store
> Implement filter pushdown for TableStoreHiveStorageHandler
> --
>
> Key: FLINK-27580
> URL: https://issues.apache.org/jira/browse/FLINK-27580
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.2.0
>
>
> Filter pushdown is a critical optimization for sources as it can decrease
> number of records to read. Hive provides a {{HiveStoragePredicateHandler}}
> interface for this purpose. We need to implement this interface in
> {{TableStoreHiveStorageHandler}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27656) Add parquet file format
[ https://issues.apache.org/jira/browse/FLINK-27656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27656: -- Component/s: Table Store > Add parquet file format > > > Key: FLINK-27656 > URL: https://issues.apache.org/jira/browse/FLINK-27656 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Priority: Major > Fix For: table-store-0.2.0 > > > The flink table store does not support parquet file format now. > Will try to publish a PR to include parquet file format in the flink table > store. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27706) Refactor all subclasses of FileStoreTableITCase to use the batchSql.
[ https://issues.apache.org/jira/browse/FLINK-27706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27706: -- Component/s: Table Store > Refactor all subclasses of FileStoreTableITCase to use the batchSql. > > > Key: FLINK-27706 > URL: https://issues.apache.org/jira/browse/FLINK-27706 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > Since we've introduced a batchSql to execute batch query for flink in > FileStoreTableITCase. Then all the subclasses can just use batch sql to > submit the flink sql. > It's a minor issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27696) Add bin-pack strategy to split the whole bucket data files into several small splits
[ https://issues.apache.org/jira/browse/FLINK-27696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27696: -- Component/s: Table Store > Add bin-pack strategy to split the whole bucket data files into several small > splits > > > Key: FLINK-27696 > URL: https://issues.apache.org/jira/browse/FLINK-27696 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > We don't have to assign each task with a whole bucket data files. Instead, we > can use some algorithm ( such as bin-packing) to split the whole bucket data > files into multiple fragments to improve the job parallelism. > For merge tree table: > Suppose now there are files: [1, 2] [3, 4] [5, 180] [5, 190] [200, 600] [210, > 700] > Files without intersection are not related, we do not need to put all files > into one split, we can slice into multiple splits, multiple parallelism > execution is faster. Nor can we slice too fine, we should make each split as > large as possible with 128 MB, so use BinPack to slice, the final result will > be: > * split1: [1, 2] [3, 4] > * split2: [5, 180] [5, 190] > * split3: [200, 600] [210, 700] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27678) Support append-only table for file store.
[ https://issues.apache.org/jira/browse/FLINK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27678: -- Component/s: Table Store > Support append-only table for file store. > - > > Key: FLINK-27678 > URL: https://issues.apache.org/jira/browse/FLINK-27678 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > Let me publish a separate PR for supporting append-only table in flink table > store's file store. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27708) Add background compaction task for append-only table when ingesting.
[ https://issues.apache.org/jira/browse/FLINK-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-27708: -- Component/s: Table Store > Add background compaction task for append-only table when ingesting. > > > Key: FLINK-27708 > URL: https://issues.apache.org/jira/browse/FLINK-27708 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Zheng Hu >Assignee: Jane Chan >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > Attachments: image-2022-06-21-14-59-59-593.png > > > We could still execute compaction task to merge small files in the background > for append-only table. > This compaction is just to avoid a lot of small files. > Its purpose is similar to that of filesystem compaction: > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#file-compaction -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-12005) [State TTL] Event time support
[ https://issues.apache.org/jira/browse/FLINK-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471675#comment-17471675 ] Yu Li commented on FLINK-12005: --- Update the ML discussion link for better history tracking > [State TTL] Event time support > -- > > Key: FLINK-12005 > URL: https://issues.apache.org/jira/browse/FLINK-12005 > Project: Flink > Issue Type: New Feature > Components: API / DataStream, Runtime / State Backends >Reporter: Andrey Zagrebin >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned > > The event time is opted for in StateTtlConfig by setting > TtlTimeCharacteristic.EventTime. > To enable event time support, the updated watermark needs to be passed to the > state backend, shared with TTL state wrappers and additional cleanup > strategies (snapshot transformers and compaction filter). > h3. Event time provider > Additional implementation of TtlTimeProvider, which holds current watermark, > needs to be passed to the state backend at the moment of its creation in > StreamTaskStateInitializerImpl. There several ways to update watermark in > this implementation of TtlTimeProvider: > * in InternalTimeServiceManager.advanceWatermark explicitly > * InternalTimeServiceManager/InternalTimerServiceImpl could be refactored to > use shared EventTimeService which holds current updatable watermark and > wrapped by TtlTimeProvider > The TTL state wrapping factory should create TTL state wrappers and snapshot > transformers with TtlTimeProvider selected by TtlTimeCharacteristic. > h3. RocksDB TTL compaction filter > The RocksDB TTL compaction filter factory needs to get selected > TtlTimeProvider when it gets configured. There are two ways: > * make it volatile and settable in > RocksDbTtlCompactFiltersManager.TimeProviderWrapper, track it in > RocksDbTtlCompactFiltersManager along with FlinkCompactionFilterFactory to > configure later before configuring FlinkCompactionFilterFactory. > * Move FlinkCompactionFilter.TimeProvider from FlinkCompactionFilterFactory > to ConfigHolder and set selected TtlTimeProvider with the Config. > The second option does not use volatile variable and should be more > performant but needs changing RocksDB java client and either releasing new > version FRocksDB or Flink RocksDB extensions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25317) ApplicationDispatcherBootstrapITCase blocked on azure
[
https://issues.apache.org/jira/browse/FLINK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-25317:
--
Fix Version/s: 1.14.3
(was: 1.14.2)
> ApplicationDispatcherBootstrapITCase blocked on azure
> -
>
> Key: FLINK-25317
> URL: https://issues.apache.org/jira/browse/FLINK-25317
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.3
>
>
> {code:java}
> Dec 15 05:25:12
> Dec 15 05:25:12 "VM Thread" os_prio=0 tid=0x7f7b641bc000 nid=0x2543b
> runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "Gang worker#0 (Parallel GC Threads)" os_prio=0
> tid=0x7f7b64020800 nid=0x25428 runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "Gang worker#1 (Parallel GC Threads)" os_prio=0
> tid=0x7f7b64022000 nid=0x2542a runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "G1 Main Concurrent Mark GC Thread" os_prio=0
> tid=0x7f7b64046000 nid=0x25431 runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0
> tid=0x7f7b64048000 nid=0x25432 runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "G1 Concurrent Refinement Thread#0" os_prio=0
> tid=0x7f7b64028000 nid=0x25430 runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "G1 Concurrent Refinement Thread#1" os_prio=0
> tid=0x7f7b64026800 nid=0x2542e runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "G1 Concurrent Refinement Thread#2" os_prio=0
> tid=0x7f7b64024800 nid=0x2542b runnable
> Dec 15 05:25:12
> Dec 15 05:25:12 "VM Periodic Task Thread" os_prio=0 tid=0x7f7b64207800
> nid=0x25446 waiting on condition
> Dec 15 05:25:12
> Dec 15 05:25:12 JNI global references: 2194
> Dec 15 05:25:12
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28133=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=9362
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20044) Disposal of RocksDB could last forever
[
https://issues.apache.org/jira/browse/FLINK-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436637#comment-17436637
]
Yu Li commented on FLINK-20044:
---
Thanks for the quick response [~wind_ljy]. Sure, let's keep watching.
> Disposal of RocksDB could last forever
> --
>
> Key: FLINK-20044
> URL: https://issues.apache.org/jira/browse/FLINK-20044
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The task cannot fail itself because it's stuck on the disposal of RocksDB,
> which also affects the job. I saw this for several times in recent months,
> most of the errors come from the broken disk. But I think we should also do
> something to deal with it more elegantly from Flink's perspective.
> {code:java}
> "LookUp_Join -> Sink_Unnamed (898/1777)- execution # 4" #411 prio=5 os_prio=0
> tid=0x7fc9b0286800 nid=0xff6fc runnable [0x7fc966cfc000]
>java.lang.Thread.State: RUNNABLE
> at org.rocksdb.RocksDB.disposeInternal(Native Method)
> at org.rocksdb.RocksObject.disposeInternal(RocksObject.java:37)
> at
> org.rocksdb.AbstractImmutableNativeReference.close(AbstractImmutableNativeReference.java:57)
> at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:263)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.dispose(RocksDBKeyedStateBackend.java:349)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.dispose(AbstractStreamOperator.java:371)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:124)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:618)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:517)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:733)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:539)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20044) Disposal of RocksDB could last forever
[
https://issues.apache.org/jira/browse/FLINK-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436616#comment-17436616
]
Yu Li commented on FLINK-20044:
---
[~wind_ljy] are we still observing the same issue in product environment? It's
a little bit stale but we will keep watching it if the later releases still
have the issue. Thanks.
> Disposal of RocksDB could last forever
> --
>
> Key: FLINK-20044
> URL: https://issues.apache.org/jira/browse/FLINK-20044
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The task cannot fail itself because it's stuck on the disposal of RocksDB,
> which also affects the job. I saw this for several times in recent months,
> most of the errors come from the broken disk. But I think we should also do
> something to deal with it more elegantly from Flink's perspective.
> {code:java}
> "LookUp_Join -> Sink_Unnamed (898/1777)- execution # 4" #411 prio=5 os_prio=0
> tid=0x7fc9b0286800 nid=0xff6fc runnable [0x7fc966cfc000]
>java.lang.Thread.State: RUNNABLE
> at org.rocksdb.RocksDB.disposeInternal(Native Method)
> at org.rocksdb.RocksObject.disposeInternal(RocksObject.java:37)
> at
> org.rocksdb.AbstractImmutableNativeReference.close(AbstractImmutableNativeReference.java:57)
> at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:263)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.dispose(RocksDBKeyedStateBackend.java:349)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.dispose(AbstractStreamOperator.java:371)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:124)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:618)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:517)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:733)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:539)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21321) Change RocksDB incremental checkpoint re-scaling to use deleteRange
[
https://issues.apache.org/jira/browse/FLINK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436506#comment-17436506
]
Yu Li commented on FLINK-21321:
---
Since we've upgraded FRocksDB version to 6.20.3 in 1.14, it's possible to
enable this feature in 1.15 (and yes let's do it), while please note that this
could only accelerate the rescaling recovery speed for scaling out but not
scaling in.
> Change RocksDB incremental checkpoint re-scaling to use deleteRange
> ---
>
> Key: FLINK-21321
> URL: https://issues.apache.org/jira/browse/FLINK-21321
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Joey Pereira
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> In FLINK-8790, it was suggested to use RocksDB's {{deleteRange}} API to more
> efficiently clip the databases for the desired target group.
> During the PR for that ticket,
> [#5582|https://github.com/apache/flink/pull/5582], the change did not end up
> using the {{deleteRange}} method as it was an experimental feature in
> RocksDB.
> At this point {{deleteRange}} is in a far less experimental state now but I
> believe is still formally "experimental". It is heavily by many others like
> CockroachDB and TiKV and they have teased out several bugs in complex
> interactions over the years.
> For certain re-scaling situations where restores trigger
> {{restoreWithScaling}} and the DB clipping logic, this would likely reduce an
> O[n] operation (N = state size/records) to O(1). For large state apps, this
> would potentially represent a non-trivial amount of time spent for
> re-scaling. In the case of my workplace, we have an operator with 100s of
> billions of records in state and re-scaling was taking a long time (>>30min,
> but it has been awhile since doing it).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-24475) Remove no longer used NestedMap* classes
[
https://issues.apache.org/jira/browse/FLINK-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned FLINK-24475:
-
Assignee: Zakelly Lan
Thanks for creating the ticket [~pnowojski], and yes this is some consensus
decision as discussed in the FLINK-21935 PR.
[~Zakelly] Please take care of this task, thanks.
> Remove no longer used NestedMap* classes
>
>
> Key: FLINK-24475
> URL: https://issues.apache.org/jira/browse/FLINK-24475
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Zakelly Lan
>Priority: Critical
> Fix For: 1.15.0
>
>
> After FLINK-21935 all of the {{NestedMapsStateTable}} classes are no longer
> used in the production code. They are still however being used in benchmarks
> in some tests. Benchmarks/tests should be migrated to {{CopyOnWrite}}
> versions while the {{NestedMaps}} classes should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23791) Enable RocksDB log again
[ https://issues.apache.org/jira/browse/FLINK-23791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-23791: -- Fix Version/s: 1.15.0 > Enable RocksDB log again > > > Key: FLINK-23791 > URL: https://issues.apache.org/jira/browse/FLINK-23791 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.14.1 > > > FLINK-15068 disabled the RocksDB's local LOG due to previous RocksDB cannot > limit the local log files. > After we upgraded to newer RocksDB version, we can then enable RocksDB log > again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13448) Support ARM architecture
[ https://issues.apache.org/jira/browse/FLINK-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406553#comment-17406553 ] Yu Li commented on FLINK-13448: --- Thanks for the feedback [~wangxiyuan], and thanks for analyzing the root cause [~bambrow]. [~bambrow] would you like to create a separate JIRA and submit a PR to fix the problem? Thanks. cc [~dian.fu] since it's PyFlink related. > Support ARM architecture > > > Key: FLINK-13448 > URL: https://issues.apache.org/jira/browse/FLINK-13448 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Stephan Ewen >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This is the umbrella issue to track the efforts to make Flink run on ARM > processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19303) Disable WAL in RocksDB recovery
[
https://issues.apache.org/jira/browse/FLINK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-19303:
--
Labels: (was: auto-deprioritized-major auto-unassigned)
> Disable WAL in RocksDB recovery
> ---
>
> Key: FLINK-19303
> URL: https://issues.apache.org/jira/browse/FLINK-19303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Juha Mynttinen
>Assignee: Juha Mynttinen
>Priority: Minor
>
> During recovery of {{RocksDBStateBackend}} the recovery mechanism puts the
> key value pairs to local RocksDB instance(s). To speed up the process, the
> recovery process uses RocskDB write batch mechanism. [RocksDB
> WAL|https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log] is enabled
> during this process.
> During normal operations, i.e. when the state backend has been recovered and
> the Flink application is running (on RocksDB state backend) WAL is disabled.
> The recovery process doesn't need WAL. In fact the recovery should be much
> faster without WAL. Thus, WAL should be disabled in the recovery process.
> AFAIK the last thing that was done with WAL during recovery was an attempt to
> remove it. Later that removal was removed because it causes stability issues
> (https://issues.apache.org/jira/browse/FLINK-8922).
> Unfortunately the root cause why disabling WAL causes segfault during
> recovery is unknown. After all, WAL is not used during normal operations.
> Potential explanation is some kind of bug in RocksDB write batch when using
> WAL. It is possible later RocksDB versions have fixes / workarounds for the
> issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-19303) Disable WAL in RocksDB recovery
[
https://issues.apache.org/jira/browse/FLINK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned FLINK-19303:
-
Assignee: Juha Mynttinen
Thanks [~juha.mynttinen], I've assigned the JIRA back to you.
> Disable WAL in RocksDB recovery
> ---
>
> Key: FLINK-19303
> URL: https://issues.apache.org/jira/browse/FLINK-19303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Juha Mynttinen
>Assignee: Juha Mynttinen
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> During recovery of {{RocksDBStateBackend}} the recovery mechanism puts the
> key value pairs to local RocksDB instance(s). To speed up the process, the
> recovery process uses RocskDB write batch mechanism. [RocksDB
> WAL|https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log] is enabled
> during this process.
> During normal operations, i.e. when the state backend has been recovered and
> the Flink application is running (on RocksDB state backend) WAL is disabled.
> The recovery process doesn't need WAL. In fact the recovery should be much
> faster without WAL. Thus, WAL should be disabled in the recovery process.
> AFAIK the last thing that was done with WAL during recovery was an attempt to
> remove it. Later that removal was removed because it causes stability issues
> (https://issues.apache.org/jira/browse/FLINK-8922).
> Unfortunately the root cause why disabling WAL causes segfault during
> recovery is unknown. After all, WAL is not used during normal operations.
> Potential explanation is some kind of bug in RocksDB write batch when using
> WAL. It is possible later RocksDB versions have fixes / workarounds for the
> issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13448) Support ARM architecture
[ https://issues.apache.org/jira/browse/FLINK-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400124#comment-17400124 ] Yu Li commented on FLINK-13448: --- [~sewen] [~wangxiyuan] From the already created sub-tasks and status, it seems only FLINK-13652 is left open (which is mainly about documentation). Does this mean functionality wise we could already fully support ARM arch? Or any other task left? Thanks. > Support ARM architecture > > > Key: FLINK-13448 > URL: https://issues.apache.org/jira/browse/FLINK-13448 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Stephan Ewen >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This is the umbrella issue to track the efforts to make Flink run on ARM > processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13652) Setup instructions for creating an ARM environment
[ https://issues.apache.org/jira/browse/FLINK-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400120#comment-17400120 ] Yu Li commented on FLINK-13652: --- Ok, maybe I'm a little bit too optimistic about the progress, and I agree that we should have some careful review before announcing formal support for ARM. Let me ask in the parent JIRA to make sure whether anyone is willing to further driving this effort. > Setup instructions for creating an ARM environment > -- > > Key: FLINK-13652 > URL: https://issues.apache.org/jira/browse/FLINK-13652 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Chesnay Schepler >Priority: Major > > We should provide developers with instructions for setting up an ARM > environment, so that they can test their changes locally without having to > rely on CI services. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14086) PrometheusReporterEndToEndITCase doesn't support ARM arch
[ https://issues.apache.org/jira/browse/FLINK-14086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned FLINK-14086: - Assignee: wangxiyuan > PrometheusReporterEndToEndITCase doesn't support ARM arch > - > > Key: FLINK-14086 > URL: https://issues.apache.org/jira/browse/FLINK-14086 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.9.0 >Reporter: wangxiyuan >Assignee: wangxiyuan >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The flink-end-to-end tests PrometheusReporterEndToEndITCase doesn't support > ARM arch. > When download prometheus package, it should get aarch64 package if the > platform is ARM64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13652) Setup instructions for creating an ARM environment
[ https://issues.apache.org/jira/browse/FLINK-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399570#comment-17399570 ] Yu Li commented on FLINK-13652: --- [~autophagy] [~chesnay] This seems to be the only sub-task left for ARM support and mainly documentation efforts, is it possible that we complete this one within the 1.14 release cycle so we could announce that in 1.14 Flink could completely support ARM arch? Or any concerns? Thanks. cc [~trohrmann] [~sewen] > Setup instructions for creating an ARM environment > -- > > Key: FLINK-13652 > URL: https://issues.apache.org/jira/browse/FLINK-13652 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Chesnay Schepler >Priority: Major > > We should provide developers with instructions for setting up an ARM > environment, so that they can test their changes locally without having to > rely on CI services. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-14482: -- Labels: pull-request-available (was: auto-deprioritized-major auto-unassigned pull-request-available) Priority: Critical (was: Minor) > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Critical > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: > Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png, > Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19303) Disable WAL in RocksDB recovery
[
https://issues.apache.org/jira/browse/FLINK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399543#comment-17399543
]
Yu Li commented on FLINK-19303:
---
[~juha.mynttinen] Since we've already upgraded FRocksDB version to 6.20.3,
would you like to restart the work here? Thanks.
> Disable WAL in RocksDB recovery
> ---
>
> Key: FLINK-19303
> URL: https://issues.apache.org/jira/browse/FLINK-19303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Juha Mynttinen
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> During recovery of {{RocksDBStateBackend}} the recovery mechanism puts the
> key value pairs to local RocksDB instance(s). To speed up the process, the
> recovery process uses RocskDB write batch mechanism. [RocksDB
> WAL|https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log] is enabled
> during this process.
> During normal operations, i.e. when the state backend has been recovered and
> the Flink application is running (on RocksDB state backend) WAL is disabled.
> The recovery process doesn't need WAL. In fact the recovery should be much
> faster without WAL. Thus, WAL should be disabled in the recovery process.
> AFAIK the last thing that was done with WAL during recovery was an attempt to
> remove it. Later that removal was removed because it causes stability issues
> (https://issues.apache.org/jira/browse/FLINK-8922).
> Unfortunately the root cause why disabling WAL causes segfault during
> recovery is unknown. After all, WAL is not used during normal operations.
> Potential explanation is some kind of bug in RocksDB write batch when using
> WAL. It is possible later RocksDB versions have fixes / workarounds for the
> issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15532) Enable strict capacity limit for memory usage for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-15532: -- Labels: pull-request-available (was: auto-deprioritized-major auto-unassigned pull-request-available) Priority: Major (was: Minor) > Enable strict capacity limit for memory usage for RocksDB > - > > Key: FLINK-15532 > URL: https://issues.apache.org/jira/browse/FLINK-15532 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: image-2020-10-23-14-39-45-997.png, > image-2020-10-23-14-41-10-584.png, image-2020-10-23-14-43-18-739.png, > image-2020-10-23-14-55-08-120.png > > > Currently, due to the limitation of RocksDB (see > [issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot > create a strict-capacity-limit LRUCache which shared among rocksDB > instance(s). > This issue tracks this problem and offer the ability of strict mode once we > could enable this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19710) Fix performance regression to rebase FRocksDB with higher version RocksDB
[
https://issues.apache.org/jira/browse/FLINK-19710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-19710:
--
Labels: (was: auto-deprioritized-major auto-unassigned)
Priority: Major (was: Minor)
> Fix performance regression to rebase FRocksDB with higher version RocksDB
> -
>
> Key: FLINK-19710
> URL: https://issues.apache.org/jira/browse/FLINK-19710
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Fix For: 1.14.0
>
>
> We planed to bump base rocksDB version from 5.17.2 to 6.11.x. However, we
> observed performance regression compared with 5.17.2 and 5.18.3 via our own
> flink-benchmarks, and reported to RocksDB community in
> [rocksdb#5774|https://github.com/facebook/rocksdb/issues/5774]. Since
> rocksDB-5.18.3 is a bit old for RocksDB community, and rocksDB built-in
> db_bench tool cannot easily reproduce this regression, we did not get any
> efficient help from RocksDB community.
> Since code freeze of Flink-release-1.12 is close, we have to figure it out by
> ourself. We try to use rocksDB built-in db_bench tool first to binary
> searching the 160 different commits between rocksDB 5.17.2 and 5.18.3.
> However, the performance regression is not so clear. And after using our own
> flink-benchmarks. We finally detect the commit which introduced the
> nearly-10% performance regression: [replaced __thread with thread_local
> keyword
> |https://github.com/facebook/rocksdb/commit/d6ec288703c8fc53b54be9e3e3f3ffd6a7487c63]
> .
> From existing knowledge, the performance regression of {{thread-local}} is
> known from [gcc-4.8 changes|https://gcc.gnu.org/gcc-4.8/changes.html#cxx] and
> become more serious in [dynamic modules usage
> |http://david-grs.github.io/tls_performance_overhead_cost_linux/] [[tls
> benchmark|https://testbit.eu/2015/thread-local-storage-benchmark]]]. That
> could explain why rocksDB built-in db_bench tool cannot reproduce this
> regression as it is complied in static mode by recommendation.
>
> We plan to fix this in our FRocksDB branch first to revert related changes.
> And from my current local experimental result, that revert proved to be
> effective to avoid that performance regression.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399497#comment-17399497 ] Yu Li edited comment on FLINK-14482 at 8/16/21, 4:39 AM: - Uploading the screenshot of micro-benchmark performance regression since the curve shown on the given links will change along with time. !Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png|width=500,height=300! !Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png|width=500,height=300! was (Author: carp84): Uploading the screenshot of micro-benchmark performance regression since the given links will change along with time. !Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png! !Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png! > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.14.0 > > Attachments: > Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png, > Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399497#comment-17399497 ] Yu Li commented on FLINK-14482: --- Uploading the screenshot of micro-benchmark performance regression since the given links will change along with time. !Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png! !Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png! > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.14.0 > > Attachments: > Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png, > Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-14482: -- Attachment: Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.14.0 > > Attachments: > Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png, > Screenshot-for-perf-regression-after-FRocksDB-upgrade-2.png > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-14482: -- Attachment: Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.14.0 > > Attachments: > Screenshot-for-perf-regression-after-FRocksDB-upgrade-1.png > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23756) Release FRocksDB-6.20.3 binaries
[ https://issues.apache.org/jira/browse/FLINK-23756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399493#comment-17399493 ] Yu Li commented on FLINK-23756: --- [~autophagy] Thanks for the efforts! How about the progress of the FRocksDB release guide update work? Could we resolve this sub-task now since the parent JIRA has already been resolved? Thanks. > Release FRocksDB-6.20.3 binaries > - > > Key: FLINK-23756 > URL: https://issues.apache.org/jira/browse/FLINK-23756 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Mika Naylor >Priority: Major > Fix For: 1.14.0 > > > Since we decided to upgrade basic RocksDB version to RocksDB-6.20.3, we need > to release FRocksDB-6.20.3 binaries. > This ticket includes work of how to release FRocksDB binaries on different > platforms and guildes for next releasing on FRocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23577) CoordinatedSourceRescaleITCase.testUpscaling fails with NoSuchFileException
[
https://issues.apache.org/jira/browse/FLINK-23577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-23577:
--
Fix Version/s: (was: 1.12.5)
1.12.6
Moved out of 1.12.5 since the RC is in progress and probably could pass. Please
feel free to move it back if the current RC is canceled and this one could be
resolved before the next one.
> CoordinatedSourceRescaleITCase.testUpscaling fails with NoSuchFileException
> ---
>
> Key: FLINK-23577
> URL: https://issues.apache.org/jira/browse/FLINK-23577
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.4
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.6
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21256=logs=298e20ef-7951-5965-0e79-ea664ddc435e=b4cd3436-dbe8-556d-3bca-42f92c3cbf2f=21306
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.874
> s <<< FAILURE! - in
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase
> [ERROR]
> testUpscaling(org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase)
> Time elapsed: 5.32 s <<< ERROR!
> java.io.UncheckedIOException: java.nio.file.NoSuchFileException:
> /tmp/junit5156435599891303309/junit3268016245125781188/79604f102e69d25f3258a72a648dfdef/chk-8
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:558)
> at
> java.base/java.util.stream.ReferencePipeline.max(ReferencePipeline.java:594)
> at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase.generateCheckpoint(CoordinatedSourceRescaleITCase.java:83)
> at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceRescaleITCase.testUpscaling(CoordinatedSourceRescaleITCase.java:70)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>
[jira] [Commented] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390330#comment-17390330
]
Yu Li commented on FLINK-23239:
---
[~lirui] Thanks for the efforts. Any reason we still haven't marked the status
of the issue as resolved? Waiting for some verification? Thanks.
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872=logs=5cae8624-c7eb-5c51-92d3-4d2dacedd221=420bd9ec-164e-562e-8947-0dacde3cec91=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23418) 'Run kubernetes application HA test' fail on Azure
[
https://issues.apache.org/jira/browse/FLINK-23418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-23418:
--
Fix Version/s: (was: 1.13.2)
1.13.3
Move the fix version from 1.13.2 to 1.13.3 since the current 1.13.2 RC3 doesn't
include this JIRA and might pass the voting and got released. Please feel free
to move it back if the current RC got canceled.
> 'Run kubernetes application HA test' fail on Azure
> --
>
> Key: FLINK-23418
> URL: https://issues.apache.org/jira/browse/FLINK-23418
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Runtime / Coordination
>Affects Versions: 1.13.1
>Reporter: Dawid Wysakowicz
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20589=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729=3747
> {code}
> Jul 16 23:58:49 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 Caused by: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka.tcp://flink@172.17.0.3:6123/user/rpc/jobmanager_2#2101744934]]
> after [1 ms]. Message of type
> [org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical reason
> for `AskTimeoutException` is that the recipient actor didn't send a reply.
> Jul 16 23:58:49 at
> akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49 at
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> Jul 16 23:58:49
[jira] [Updated] (FLINK-23315) Bump log4j to 2.14.1 for version 1.13.2
[ https://issues.apache.org/jira/browse/FLINK-23315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-23315: -- Fix Version/s: (was: 1.13.2) 1.13.3 Change fix version to 1.13.3 since we're checking against 1.13.2 RC3 and this issue is still not assigned. Please feel free to set it back to 1.13.2 if 1.13.2 RC3 is canceled and this issue could be resolved before the next RC. > Bump log4j to 2.14.1 for version 1.13.2 > --- > > Key: FLINK-23315 > URL: https://issues.apache.org/jira/browse/FLINK-23315 > Project: Flink > Issue Type: Improvement >Reporter: Guilaume Kermorgant >Priority: Minor > Fix For: 1.13.3 > > > Flink 1.13 is currently [relying on log4j 2.12.1|#L110], which has a [low > severity vulnerability|[https://nvd.nist.gov/vuln/detail/CVE-2020-9488]]. > This is fixed in Log4j 2.13.1. > Flink 1.14 will be released with Log4j 2.14.1, c.f. FLINK-22407 > It would be nice for us to have it in Flink 1.13.2 as well, if the community > thinks it's not a bad idea; this could also be a good opportunity for me to > open a first PR in the Flink repo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383180#comment-17383180 ] Yu Li commented on FLINK-14482: --- Sorry for the late response [~trohrmann], just noticed... [~yunta] is still trying to locate the root cause of performance regression in 6.x as mentioned in [the above comment|https://issues.apache.org/jira/browse/FLINK-14482?focusedCommentId=17228997=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17228997], or counteracting the regression by leveraging some new features in higher version, and we are not 100% sure whether we could make it in 1.14 release. However, we will try our best and update here if any progress. > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.14.0 > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-21831) Add Changelog state for timers (PQ)
[ https://issues.apache.org/jira/browse/FLINK-21831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned FLINK-21831: - Assignee: Piotr Nowojski > Add Changelog state for timers (PQ) > > > Key: FLINK-21831 > URL: https://issues.apache.org/jira/browse/FLINK-21831 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-22678) Hide ChangelogStateBackend From Users
[ https://issues.apache.org/jira/browse/FLINK-22678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned FLINK-22678: - Assignee: Zakelly Lan (was: Yuan Mei) > Hide ChangelogStateBackend From Users > -- > > Key: FLINK-22678 > URL: https://issues.apache.org/jira/browse/FLINK-22678 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Yuan Mei >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > Per discussion on the mailing thread: > https://lists.apache.org/thread.html/ra178ce29088b1da362d98a5a6d8c7be48051caf1637ee24261738217%40%3Cdev.flink.apache.org%3E > We decide to make a refined version of loading ChangelogStateBackend: > - Define consistent override and combination policy (flag + state backend) > in different config levels > - Define explicitly the meaning of "enable flag" = true/false/unset > - Hide ChangelogStateBackend from users > Details described in > https://docs.google.com/document/d/13AaCf5fczYTDHZ4G1mgYL685FqbnoEhgo0cdwuJlZmw/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-21804) Create and wire changelog writer with backend
[
https://issues.apache.org/jira/browse/FLINK-21804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned FLINK-21804:
-
Assignee: Zakelly Lan
Thanks for volunteering [~Zakelly]! Just assigned to you.
> Create and wire changelog writer with backend
> -
>
> Key: FLINK-21804
> URL: https://issues.apache.org/jira/browse/FLINK-21804
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Roman Khachatryan
>Assignee: Zakelly Lan
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: Changelog Backend _ writer loading.png
>
>
> [Proposed
> design|https://docs.google.com/document/d/10c6hZsOVxzUjeCLPSDpKGyZOYHi73yd92lCqRs1CyUE/edit#heading=h.5b9hthjg53vl]
> !Changelog Backend _ writer loading.png|width=600!
> * Black arrows - existing references/creations
> * {color:red}Red{color} arrows - required references/creations
> * {color:#00875A}Green{color} arrows - proposed references/creations to
> enable required ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-5601) Window operator does not checkpoint watermarks
[
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-5601:
-
Labels: pull-request-available (was: pull-request-available stale-assigned)
Thanks for the quick response [~wind_ljy], let me remove the stale-assigned
label first.
And please feel free to start a ML discussion if that could help move the work
forward.
> Window operator does not checkpoint watermarks
> --
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.11.0
>Reporter: Ufuk Celebi
>Assignee: Jiayi Liao
>Priority: Critical
> Labels: pull-request-available
>
> During release testing [~stefanrichte...@gmail.com] and I noticed that
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and
> rescaling and ran into failures, because the data generator required
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not
> dropped, because the watermarks needed to be re-established after restore
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not
> checkpointing watermarks?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-15507: -- Labels: pull-request-available (was: pull-request-available stale-assigned) This is something we're still actively tracking. > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Assignee: Zakelly Lan >Priority: Critical > Labels: pull-request-available > Fix For: 1.14.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19303) Disable WAL in RocksDB recovery
[
https://issues.apache.org/jira/browse/FLINK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-19303:
--
Labels: (was: stale-assigned)
> Disable WAL in RocksDB recovery
> ---
>
> Key: FLINK-19303
> URL: https://issues.apache.org/jira/browse/FLINK-19303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Juha Mynttinen
>Assignee: Juha Mynttinen
>Priority: Major
>
> During recovery of {{RocksDBStateBackend}} the recovery mechanism puts the
> key value pairs to local RocksDB instance(s). To speed up the process, the
> recovery process uses RocskDB write batch mechanism. [RocksDB
> WAL|https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log] is enabled
> during this process.
> During normal operations, i.e. when the state backend has been recovered and
> the Flink application is running (on RocksDB state backend) WAL is disabled.
> The recovery process doesn't need WAL. In fact the recovery should be much
> faster without WAL. Thus, WAL should be disabled in the recovery process.
> AFAIK the last thing that was done with WAL during recovery was an attempt to
> remove it. Later that removal was removed because it causes stability issues
> (https://issues.apache.org/jira/browse/FLINK-8922).
> Unfortunately the root cause why disabling WAL causes segfault during
> recovery is unknown. After all, WAL is not used during normal operations.
> Potential explanation is some kind of bug in RocksDB write batch when using
> WAL. It is possible later RocksDB versions have fixes / workarounds for the
> issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19303) Disable WAL in RocksDB recovery
[
https://issues.apache.org/jira/browse/FLINK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324785#comment-17324785
]
Yu Li commented on FLINK-19303:
---
Marking as blocked by FLINK-14482
> Disable WAL in RocksDB recovery
> ---
>
> Key: FLINK-19303
> URL: https://issues.apache.org/jira/browse/FLINK-19303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Juha Mynttinen
>Assignee: Juha Mynttinen
>Priority: Major
> Labels: stale-assigned
>
> During recovery of {{RocksDBStateBackend}} the recovery mechanism puts the
> key value pairs to local RocksDB instance(s). To speed up the process, the
> recovery process uses RocskDB write batch mechanism. [RocksDB
> WAL|https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log] is enabled
> during this process.
> During normal operations, i.e. when the state backend has been recovered and
> the Flink application is running (on RocksDB state backend) WAL is disabled.
> The recovery process doesn't need WAL. In fact the recovery should be much
> faster without WAL. Thus, WAL should be disabled in the recovery process.
> AFAIK the last thing that was done with WAL during recovery was an attempt to
> remove it. Later that removal was removed because it causes stability issues
> (https://issues.apache.org/jira/browse/FLINK-8922).
> Unfortunately the root cause why disabling WAL causes segfault during
> recovery is unknown. After all, WAL is not used during normal operations.
> Potential explanation is some kind of bug in RocksDB write batch when using
> WAL. It is possible later RocksDB versions have fixes / workarounds for the
> issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15532) Enable strict capacity limit for memory usage for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-15532: -- Labels: (was: stale-assigned) > Enable strict capacity limit for memory usage for RocksDB > - > > Key: FLINK-15532 > URL: https://issues.apache.org/jira/browse/FLINK-15532 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Attachments: image-2020-10-23-14-39-45-997.png, > image-2020-10-23-14-41-10-584.png, image-2020-10-23-14-43-18-739.png, > image-2020-10-23-14-55-08-120.png > > > Currently, due to the limitation of RocksDB (see > [issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot > create a strict-capacity-limit LRUCache which shared among rocksDB > instance(s). > This issue tracks this problem and offer the ability of strict mode once we > could enable this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15532) Enable strict capacity limit for memory usage for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324781#comment-17324781 ] Yu Li commented on FLINK-15532: --- This is still something in the plan and currently blocked by FLINK-14482. > Enable strict capacity limit for memory usage for RocksDB > - > > Key: FLINK-15532 > URL: https://issues.apache.org/jira/browse/FLINK-15532 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: stale-assigned > Attachments: image-2020-10-23-14-39-45-997.png, > image-2020-10-23-14-41-10-584.png, image-2020-10-23-14-43-18-739.png, > image-2020-10-23-14-55-08-120.png > > > Currently, due to the limitation of RocksDB (see > [issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot > create a strict-capacity-limit LRUCache which shared among rocksDB > instance(s). > This issue tracks this problem and offer the ability of strict mode once we > could enable this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15424) Make all AppendingState#add respect the java doc
[
https://issues.apache.org/jira/browse/FLINK-15424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324774#comment-17324774
]
Yu Li commented on FLINK-15424:
---
[~klion26] are you still actively working on this one? Thanks.
> Make all AppendingState#add respect the java doc
>
>
> Key: FLINK-15424
> URL: https://issues.apache.org/jira/browse/FLINK-15424
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / State Backends
>Affects Versions: 1.8.3, 1.9.1
>Reporter: Congxian Qiu
>Assignee: Congxian Qiu
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, We have a java doc in
> {{[AppendingState#add|https://github.com/apache/flink/blob/52fdee1d0c7af24d25c51caa073e29f11b07210b/flink-core/src/main/java/org/apache/flink/api/common/state/AppendingState.java#L63]}}
> {code:java}
> If null is passed in, the state value will remain unchanged.{code}
> but currently, the implementation did not respect this, take
> {{HeapReducingState}} as an example, we'll clear the state if the passed
> parameter is null
> {code:java}
> @Override
> public void add(V value) throws IOException {
> if (value == null) {
> clear();
> return;
> }
> try {
> stateTable.transform(currentNamespace, value, reduceTransformation);
> } catch (Exception e) {
> throw new IOException("Exception while applying ReduceFunction in
> reducing state", e);
> }
> }
> {code}
> But in {{RocksDBReducingState}} we would not clear the state, and put the
> null value into state if serializer can serialize null.
> {code:java}
> @Override
> public void add(V value) throws Exception {
>byte[] key = getKeyBytes();
>V oldValue = getInternal(key);
>V newValue = oldValue == null ? value : reduceFunction.reduce(oldValue,
> value);
>updateInternal(key, newValue);
> }
> {code}
> this issue wants to make all {{Appending}}State respect the javadoc of
> {{AppendingState}}, and return directly if the passed in parameter is null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15012) Checkpoint directory not cleaned up
[
https://issues.apache.org/jira/browse/FLINK-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324765#comment-17324765
]
Yu Li commented on FLINK-15012:
---
I guess we cannot make this in 1.13 already and need to postpone to 1.14?
Please update the JIRA accordingly [~yunta]
> Checkpoint directory not cleaned up
> ---
>
> Key: FLINK-15012
> URL: https://issues.apache.org/jira/browse/FLINK-15012
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.1
>Reporter: Nico Kruber
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I started a Flink cluster with 2 TMs using {{start-cluster.sh}} and the
> following config (in addition to the default {{flink-conf.yaml}})
> {code:java}
> state.checkpoints.dir: file:///path/to/checkpoints/
> state.backend: rocksdb {code}
> After submitting a jobwith checkpoints enabled (every 5s), checkpoints show
> up, e.g.
> {code:java}
> bb969f842bbc0ecc3b41b7fbe23b047b/
> ├── chk-2
> │ ├── 238969e1-6949-4b12-98e7-1411c186527c
> │ ├── 2702b226-9cfc-4327-979d-e5508ab2e3d5
> │ ├── 4c51cb24-6f71-4d20-9d4c-65ed6e826949
> │ ├── e706d574-c5b2-467a-8640-1885ca252e80
> │ └── _metadata
> ├── shared
> └── taskowned {code}
> If I shut down the cluster via {{stop-cluster.sh}}, these files will remain
> on disk and not be cleaned up.
> In contrast, if I cancel the job, at least {{chk-2}} will be deleted, but
> still leaving the (empty) directories.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-13194) Add explicit clarification about thread-safety of state in document
[ https://issues.apache.org/jira/browse/FLINK-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned FLINK-13194: - Assignee: Yun Tang (was: Yu Li) Labels: pull-request-available (was: pull-request-available stale-assigned) This is still something we should improve. Re-assign to [~yunta] who is actually working on it. > Add explicit clarification about thread-safety of state in document > --- > > Key: FLINK-13194 > URL: https://issues.apache.org/jira/browse/FLINK-13194 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / State Backends >Reporter: Yu Li >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > > Currently there's no explicit clarification about whether state is thread > safe in our document meanwhile there are many user questions about this such > as FLINK-13072, on > [stackoverflow|https://stackoverflow.com/questions/55208345/flink-operator-state-is-thread-safe] > and in [mailing list|https://s.apache.org/ya10u], and this JIRA aims at > improving this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14032) Make the cache size of RocksDBPriorityQueueSetFactory configurable
[
https://issues.apache.org/jira/browse/FLINK-14032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-14032:
--
Fix Version/s: (was: 1.13.0)
1.14.0
Labels: usability (was: stale-assigned usability)
The TODO flag still lies there and let's get the work done in 1.14.0 [~yunta]
> Make the cache size of RocksDBPriorityQueueSetFactory configurable
> --
>
> Key: FLINK-14032
> URL: https://issues.apache.org/jira/browse/FLINK-14032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
> Labels: usability
> Fix For: 1.14.0
>
>
> Currently, the cache size of {{RocksDBPriorityQueueSetFactory}} has been set
> as 128 and no any ways to configure this to other value. (We could increase
> this to obtain better performance if necessary). Actually, this is also a
> TODO for quiet a long time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-14482: -- Labels: pull-request-available (was: pull-request-available stale-assigned) This is something we're still actively tracking and currently blocked by FLINK-19710 > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19710) Fix performance regression to rebase FRocksDB with higher version RocksDB
[
https://issues.apache.org/jira/browse/FLINK-19710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-19710:
--
Labels: (was: stale-assigned)
This is something we're still actively tracking.
> Fix performance regression to rebase FRocksDB with higher version RocksDB
> -
>
> Key: FLINK-19710
> URL: https://issues.apache.org/jira/browse/FLINK-19710
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Fix For: 1.14.0
>
>
> We planed to bump base rocksDB version from 5.17.2 to 6.11.x. However, we
> observed performance regression compared with 5.17.2 and 5.18.3 via our own
> flink-benchmarks, and reported to RocksDB community in
> [rocksdb#5774|https://github.com/facebook/rocksdb/issues/5774]. Since
> rocksDB-5.18.3 is a bit old for RocksDB community, and rocksDB built-in
> db_bench tool cannot easily reproduce this regression, we did not get any
> efficient help from RocksDB community.
> Since code freeze of Flink-release-1.12 is close, we have to figure it out by
> ourself. We try to use rocksDB built-in db_bench tool first to binary
> searching the 160 different commits between rocksDB 5.17.2 and 5.18.3.
> However, the performance regression is not so clear. And after using our own
> flink-benchmarks. We finally detect the commit which introduced the
> nearly-10% performance regression: [replaced __thread with thread_local
> keyword
> |https://github.com/facebook/rocksdb/commit/d6ec288703c8fc53b54be9e3e3f3ffd6a7487c63]
> .
> From existing knowledge, the performance regression of {{thread-local}} is
> known from [gcc-4.8 changes|https://gcc.gnu.org/gcc-4.8/changes.html#cxx] and
> become more serious in [dynamic modules usage
> |http://david-grs.github.io/tls_performance_overhead_cost_linux/] [[tls
> benchmark|https://testbit.eu/2015/thread-local-storage-benchmark]]]. That
> could explain why rocksDB built-in db_bench tool cannot reproduce this
> regression as it is complied in static mode by recommendation.
>
> We plan to fix this in our FRocksDB branch first to revert related changes.
> And from my current local experimental result, that revert proved to be
> effective to avoid that performance regression.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-5601) Window operator does not checkpoint watermarks
[
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324741#comment-17324741
]
Yu Li commented on FLINK-5601:
--
I believe this is still something we need to follow up. [~wind_ljy] are you
still working on this? Thanks.
> Window operator does not checkpoint watermarks
> --
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.11.0
>Reporter: Ufuk Celebi
>Assignee: Jiayi Liao
>Priority: Critical
> Labels: pull-request-available, stale-assigned
>
> During release testing [~stefanrichte...@gmail.com] and I noticed that
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and
> rescaling and ran into failures, because the data generator required
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not
> dropped, because the watermarks needed to be re-established after restore
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not
> checkpointing watermarks?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-10954) Hardlink from files of previous local stored state might cross devices
[
https://issues.apache.org/jira/browse/FLINK-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-10954:
--
Labels: (was: stale-assigned)
This is something we still need to fix, although the priority seems to be not
that high.
> Hardlink from files of previous local stored state might cross devices
> --
>
> Key: FLINK-10954
> URL: https://issues.apache.org/jira/browse/FLINK-10954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.6.2
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Critical
>
> Currently, local recovery's base directories is initialized from
> '{{io.tmp.dirs}}' if parameter '{{taskmanager.state.local.root-dirs}}' is not
> set. For Yarn environment, the tmp dirs is replaced by its '{{LOCAL_DIRS}}',
> which might consist of directories from different devices, such as
> /dump/1/nm-local-dir, /dump/2/nm-local-dir. The local directory for RocksDB
> is initialized from IOManager's spillingDirectories, which might located in
> different device from local recovery's folder. However, hard-link between
> different devices is not allowed, it will throw exception below:
> {code:java}
> java.nio.file.FileSystemException: target -> souce: Invalid cross-device link
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-11254) Unify serialization format of savepoint for switching state backends
[ https://issues.apache.org/jira/browse/FLINK-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li closed FLINK-11254. - Resolution: Duplicate This feature is already implemented by FLINK_20976, closing as duplicate. > Unify serialization format of savepoint for switching state backends > > > Key: FLINK-11254 > URL: https://issues.apache.org/jira/browse/FLINK-11254 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.7.1 >Reporter: Congxian Qiu >Assignee: Congxian Qiu >Priority: Major > Labels: stale-assigned > > For the current version, the serialization formats of savepoint between > HeapKeyedStateBackend and RocksDBStateBackend are different, so we can not > switch state backend when using savepoint. We should unify the serialization > formats of the savepoint to support state backend switch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13251) Add bandwidth throttling for checkpoint uploading/downloading
[ https://issues.apache.org/jira/browse/FLINK-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-13251: -- Labels: (was: stale-assigned) I think this is still nice to have but I won't be able to work on it in the near future. I'm removing the stale-assigned label and just let me know if anyone would like to take it over, will do the re-assign and help on PR review. > Add bandwidth throttling for checkpoint uploading/downloading > - > > Key: FLINK-13251 > URL: https://issues.apache.org/jira/browse/FLINK-13251 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > > As > [reported|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Bandwidth-throttling-of-checkpoints-uploading-to-s3-tt28735.html] > in our user mailing list, the checkpoint uploading may make high load to the > network. In contrast to accelerating checkpoint downloading/uploading as > introduced by FLINK-10461/FLINK-11008, I think it also makes sense to add a > feature to allow bandwidth throttling, and this JIRA aims at introducing this > feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13815) Implement the SpaceAllocator
[
https://issues.apache.org/jira/browse/FLINK-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13815:
--
Labels: pull-request-available (was: pull-request-available stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Implement the SpaceAllocator
>
>
> Key: FLINK-13815
> URL: https://issues.apache.org/jira/browse/FLINK-13815
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As described in the design doc, we need a {{SpaceAllocator}} to allocate
> space on off-heap/disk to store the spilled key-group data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13803) Introduce SpillableHeapKeyedStateBackend and all necessities
[
https://issues.apache.org/jira/browse/FLINK-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13803:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Introduce SpillableHeapKeyedStateBackend and all necessities
>
>
> Key: FLINK-13803
> URL: https://issues.apache.org/jira/browse/FLINK-13803
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: PengFei Li
>Priority: Major
>
> This JIRA aims at introducing a new {{SpillableHeapKeyedStateBackend}} which
> will reuse most code of the {{HeapKeyedStateBackend}} (probably the only
> difference is the spill-able one will register a {{HybridStateTable}}), and
> allow using it in {{FsStateBackend}} and {{MemoryStateBackend}} (only as an
> option, by default still {{HeapKeyedStateBackend}}) through configuration.
> The related necessities include but are not limited to:
> * A relative backend builder class
> * Relative restore operation classes
> * Necessary configurations for using spill-able backend
> This should be the last JIRA after which the spill-able heap backend feature
> will become runnable regardless of the stability and performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13801) Introduce a HybridStateTable to combine everything together
[
https://issues.apache.org/jira/browse/FLINK-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13801:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Introduce a HybridStateTable to combine everything together
> ---
>
> Key: FLINK-13801
> URL: https://issues.apache.org/jira/browse/FLINK-13801
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: PengFei Li
>Priority: Major
>
> This JIRA aims at introducing a {{HybridStateTable}} which could combine
> everything together, like checking the heap usage through
> {{HeapAccountingManager}} and GC status through {{HeapStatusMonitor}} to
> decide whether to spill/load key groups, triggering the spill/load action
> through {{SpillLoadManager}}, and recording all meta data (about which key
> group is on heap, which is on disk), etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12698) Implement a SpillLoadManager
[
https://issues.apache.org/jira/browse/FLINK-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-12698:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Implement a SpillLoadManager
>
>
> Key: FLINK-12698
> URL: https://issues.apache.org/jira/browse/FLINK-12698
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
>
> The {{SpillLoadManager}} is responsible to:
> 1. Do spill/load when {{HeapStatusMonitor}} triggers.
> 2. Decide which KeyGroup to spill/load according to data from the accounting
> managers (mainly two factors: state-size and request-rate).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12696) Implement a MmapManager
[
https://issues.apache.org/jira/browse/FLINK-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-12696:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Implement a MmapManager
> ---
>
> Key: FLINK-12696
> URL: https://issues.apache.org/jira/browse/FLINK-12696
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
>
> Since we plan to use mmap to accelerate the state read/write against the
> spilled (on-disk) KeyGroup, we need a {{MmapManager}} to manage the mmap-ed
> files, pretty much like the
> [MemoryMappedFileManager|https://s.apache.org/85BP] in log4j2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12699) Reduce CPU consumption when snapshot/restore the spilled key-group
[
https://issues.apache.org/jira/browse/FLINK-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-12699:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Reduce CPU consumption when snapshot/restore the spilled key-group
> --
>
> Key: FLINK-12699
> URL: https://issues.apache.org/jira/browse/FLINK-12699
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: PengFei Li
>Priority: Major
>
> We need to prevent the unnecessary de/serialization when
> snapshotting/restoring the spilled state key-group. To achieve this, we need
> to:
> 1. Add meta information for {{HeapKeyedStatebackend}} checkpoint on DFS,
> separating the on-heap and on-disk part
> 2. Write the off-heap bytes directly to DFS when checkpointing and mark it as
> on-disk
> 3. Directly write the bytes onto disk when restoring the data back from DFS,
> if it's marked as on-disk
> Notice that we cannot directly use file copy since we use mmap meanwhile
> support copy-on-write.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12695) Implement a HeapStatusMonitor
[
https://issues.apache.org/jira/browse/FLINK-12695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-12695:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Implement a HeapStatusMonitor
> -
>
> Key: FLINK-12695
> URL: https://issues.apache.org/jira/browse/FLINK-12695
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
>
> We need a {{HeapStatusMonitor}} to monitor the JVM heap status and do
> necessary spill/load when heap is exhausted/regained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12694) Implement a HeapAccountingManager
[
https://issues.apache.org/jira/browse/FLINK-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-12694:
--
Labels: (was: stale-assigned)
The work was put into backlog due to priority and lack of review resource but
still not abandoned.
> Implement a HeapAccountingManager
> -
>
> Key: FLINK-12694
> URL: https://issues.apache.org/jira/browse/FLINK-12694
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
>
> We need a {{HeapAccountingManager}} for a) (roughly) estimating the on-heap
> size of each key-value; and b) recording the size of each on-heap and
> off-heap KeyGroup. With such accountings we could decide which KeyGroup to
> spill/load.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20496) RocksDB partitioned index filter option
[
https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317703#comment-17317703
]
Yu Li commented on FLINK-20496:
---
[~maver1ck] Sorry that somehow I missed the ping and just noticed the message...
Normally we will only backport bug fix and operational improvements for minor
releases, and I think we could regard this one as the later case since it helps
to resolve perf-related (operational) problems. Would you like to open a new
JIRA and submit a PR against release-1.12 for the backport (since you've
already done this offline)? I will be there to help review and merge the
changes (smile).
And glad to know this work helps in your real-world workloads!
> RocksDB partitioned index filter option
> ---
>
> Key: FLINK-20496
> URL: https://issues.apache.org/jira/browse/FLINK-20496
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.10.2, 1.11.2, 1.12.0
>Reporter: YufeiLiu
>Assignee: YufeiLiu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> When using RocksDBStateBackend and enabling
> {{state.backend.rocksdb.memory.managed}} and
> {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited
> rocksdb memory usage which contains "write buffer" and "block cache". With
> these options rocksdb stores index and filters in block cache, because in
> default options index/filters can grows unlimited.
> But it's lead another issue, if high-priority cache(configure by
> {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all
> index/filters blocks, it will load all metadata from disk when cache missed,
> and program went extremely slow. According to [Partitioned Index
> Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1],
> we can enable two-level index having acceptable performance when
> index/filters cache missed.
> Enable these options can get over 10x faster in my case[2], I think we can
> add an option {{state.backend.rocksdb.partitioned-index-filters}} and default
> value is false, so we can use this feature easily.
> [1] Partitioned Index Filters:
> https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M,
> SSD, elapsed time 4.91ms -> 0.33ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20496) RocksDB partitioned index filter option
[
https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317697#comment-17317697
]
Yu Li commented on FLINK-20496:
---
[~sewen] According to the RocksDB
[document|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters#cons],
we could find below descriptions about constraints of partitioned
index/filters:
* Additional space for the top-level index: its quite small 0.1-1% of
index/filter size.
* More disk IO: if the top-level index is not already in cache it would result
to one additional IO. To avoid that they can be either stored in heap or stored
in cache with hi priority
* Losing spatial locality: if a workload requires frequent, yet random reads
from the same SST file, it would result into loading a separate index/filter
partition upon each read, which is less efficient than reading the entire
index/filter at once. Although we did not observe this pattern in our
benchmarks, it is only likely to happen for L0/L1 layers of LSM, for which
partitioning can be disabled (TODO work)
Since [by
default|https://github.com/apache/flink/blob/b17e7b5d3504a4b70a9c9bcf50175a235e9186fe/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBResourceContainer.java#L162]
we will cache index and filter blocks with high priority, I think we're ok
with the first two aspects. For the third one, it seems only affecting
workloads with hotspots.
Personally I'm optimistic about turning this on by default, but would suggest
to be more cautious and do more testing against our
[micro-benchmarks|https://github.com/apache/flink-benchmarks] and stateful
cases in [nexmark|https://github.com/nexmark/nexmark].
Please let me know your thoughts [~sewen] [~liuyufei]. Thanks.
> RocksDB partitioned index filter option
> ---
>
> Key: FLINK-20496
> URL: https://issues.apache.org/jira/browse/FLINK-20496
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.10.2, 1.11.2, 1.12.0
>Reporter: YufeiLiu
>Assignee: YufeiLiu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> When using RocksDBStateBackend and enabling
> {{state.backend.rocksdb.memory.managed}} and
> {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited
> rocksdb memory usage which contains "write buffer" and "block cache". With
> these options rocksdb stores index and filters in block cache, because in
> default options index/filters can grows unlimited.
> But it's lead another issue, if high-priority cache(configure by
> {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all
> index/filters blocks, it will load all metadata from disk when cache missed,
> and program went extremely slow. According to [Partitioned Index
> Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1],
> we can enable two-level index having acceptable performance when
> index/filters cache missed.
> Enable these options can get over 10x faster in my case[2], I think we can
> add an option {{state.backend.rocksdb.partitioned-index-filters}} and default
> value is false, so we can use this feature easily.
> [1] Partitioned Index Filters:
> https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M,
> SSD, elapsed time 4.91ms -> 0.33ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21694) Increase default value of "state.backend.rocksdb.checkpoint.transfer.thread.num"
[ https://issues.apache.org/jira/browse/FLINK-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316939#comment-17316939 ] Yu Li commented on FLINK-21694: --- btw, my JIRA id is `liyu` and `liyu2` is another person, so I didn't notice this JIRA at first, JFYI [~sewen] > Increase default value of > "state.backend.rocksdb.checkpoint.transfer.thread.num" > > > Key: FLINK-21694 > URL: https://issues.apache.org/jira/browse/FLINK-21694 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Priority: Critical > Fix For: 1.13.0 > > > The default value for the number of threads used to download state artifacts > from checkpoint storage should be increased. > The increase should not pose risk of regression, but does in many cases speed > up checkpoint recovery significantly. > Something similar was reported in this blog post, item (3). > https://engineering.contentsquare.com/2021/ten-flink-gotchas/ > A default value of 8 (eight) sounds like a good default. It should not result > in excessive thread explosion, and already speeds up recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21694) Increase default value of "state.backend.rocksdb.checkpoint.transfer.thread.num"
[ https://issues.apache.org/jira/browse/FLINK-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316934#comment-17316934 ] Yu Li commented on FLINK-21694: --- +1 to increase the default value of `state.backend.rocksdb.checkpoint.transfer.thread.num`. Regarding the detailed value to increase to, I agree that 4 is safer for those big clusters with massive stateful jobs. As to whether to change the fault value for `state.backend.rocksdb.thread.num`, I suggest to open another JIRA for discussion and keep this one focused. > Increase default value of > "state.backend.rocksdb.checkpoint.transfer.thread.num" > > > Key: FLINK-21694 > URL: https://issues.apache.org/jira/browse/FLINK-21694 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Priority: Critical > Fix For: 1.13.0 > > > The default value for the number of threads used to download state artifacts > from checkpoint storage should be increased. > The increase should not pose risk of regression, but does in many cases speed > up checkpoint recovery significantly. > Something similar was reported in this blog post, item (3). > https://engineering.contentsquare.com/2021/ten-flink-gotchas/ > A default value of 8 (eight) sounds like a good default. It should not result > in excessive thread explosion, and already speeds up recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-15507: -- Fix Version/s: (was: 1.13.0) 1.14.0 Thanks for checking [~dwysakowicz]. I'm afraid we cannot complete this in 1.13 and let's postpone it to 1.14. Fix version updated. > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Assignee: Zakelly Lan >Priority: Critical > Labels: pull-request-available > Fix For: 1.14.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend
[
https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li closed FLINK-21413.
-
Fix Version/s: 1.13.0
Resolution: Fixed
Merged into master via 1e17621e65bfa8f4f3aff379118654ed77a82bd2
> TtlMapState and TtlListState cannot be clean completely with Filesystem
> StateBackend
>
>
> Key: FLINK-21413
> URL: https://issues.apache.org/jira/browse/FLINK-21413
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Assignee: Jiayi Liao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Attachments: image-2021-02-19-11-13-58-672.png
>
>
> Take the #TtlMapState as an example,
>
> {code:java}
> public Map> getUnexpiredOrNull(@Nonnull Map TtlValue> ttlValue) {
> Map> unexpired = new HashMap<>();
> TypeSerializer> valueSerializer =
> ((MapSerializer>)
> original.getValueSerializer()).getValueSerializer();
> for (Map.Entry> e : ttlValue.entrySet()) {
> if (!expired(e.getValue())) {
> // we have to do the defensive copy to update the
> value
> unexpired.put(e.getKey(),
> valueSerializer.copy(e.getValue()));
> }
> }
> return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
> }
> {code}
>
> The returned value will never be null and the #StateEntry will exists
> forever, which leads to memory leak if the key's range of the stream is very
> large. Below we can see that 20+ millison uncleared TtlStateMap could take up
> several GB memory.
>
> !image-2021-02-19-11-13-58-672.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-22004) Translate Flink Roadmap to Chinese.
[ https://issues.apache.org/jira/browse/FLINK-22004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned FLINK-22004: - Assignee: Yuan Mei Thanks for following this up [~ym], assigned to you. > Translate Flink Roadmap to Chinese. > --- > > Key: FLINK-22004 > URL: https://issues.apache.org/jira/browse/FLINK-22004 > Project: Flink > Issue Type: Task > Components: Documentation >Reporter: Yuan Mei >Assignee: Yuan Mei >Priority: Major > > https://flink.apache.org/roadmap.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18712) Flink RocksDB statebackend memory leak issue
[ https://issues.apache.org/jira/browse/FLINK-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309526#comment-17309526 ] Yu Li commented on FLINK-18712: --- [~yunta] Maybe some description of the newly attached demo package? Thanks. > Flink RocksDB statebackend memory leak issue > - > > Key: FLINK-18712 > URL: https://issues.apache.org/jira/browse/FLINK-18712 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.10.0 >Reporter: Farnight >Assignee: Yun Tang >Priority: Critical > Labels: usability > Attachments: flink-demo-master.tgz > > > When using RocksDB as our statebackend, we found it will lead to memory leak > when restarting job (manually or in recovery case). > > How to reproduce: > # increase RocksDB blockcache size(e.g. 1G), it is easier to monitor and > reproduce. > # start a job using RocksDB statebackend. > # when the RocksDB blockcache reachs maximum size, restart the job. and > monitor the memory usage (k8s pod working set) of the TM. > # go through step 2-3 few more times. and memory will keep raising. > > Any solution or suggestion for this? Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21935) Remove "state.backend.async" option.
[
https://issues.apache.org/jira/browse/FLINK-21935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307562#comment-17307562
]
Yu Li commented on FLINK-21935:
---
+1 to remove the option and synchronous snapshot support for heap backend.
Shall we send a notice in our user mailing list or release note is enough?
Since the default value of {{state.backend.async}} is {{true}}, I believe the
use of synchronous snapshot is pretty rare.
I could see more related works to do, such as deprecating
{{NestedMapsStateTable}} and do clean ups in the next release (or maybe we
could carry this out in this release since if we remove the option then no way
to use it after all?).
I will be ready to do the PR review, just let me know (smile)
> Remove "state.backend.async" option.
>
>
> Key: FLINK-21935
> URL: https://issues.apache.org/jira/browse/FLINK-21935
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.13.0
>
>
> Checkpoints are always asynchronous, there is no case ever for a synchronous
> checkpoint.
> The RocksDB state backend doesn't even support synchronous snapshots, and the
> HashMap Heap backend also has no good use case for synchronous snapshots
> (other than a very minor reduction in heap objects).
> Most importantly, we should not expose this option in the constructors of the
> new state backend API classes, like {{HashMapStateBackend}}.
> I marked this a blocker because it is part of the new user-facing State
> Backend API and I would like to avoid that this option enters this API and
> causes confusion when we eventually remove it.
> /cc [~sjwiesman] and [~liyu]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21695) Increase default value for number of KeyGroups
[
https://issues.apache.org/jira/browse/FLINK-21695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17298572#comment-17298572
]
Yu Li commented on FLINK-21695:
---
Thanks for the note [~sewen], I also noticed the blog through twitter and
totally agree to do some software (instead of operational) level improvement.
One input for the fix, that in our internal version (blink) we chose the first
way (set default value to 32768), and the change was applied from very early
days (when I was even not working on blink yet). Maybe [~maguowei] could give
some more information about why we didn't choose the second way.
How to keep backward compatibility (restoring from old
savepoint/retained-checkpoint) is indeed a problem, and I don't have any better
idea yet...
> Increase default value for number of KeyGroups
> --
>
> Key: FLINK-21695
> URL: https://issues.apache.org/jira/browse/FLINK-21695
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / State Backends
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 1.13.0
>
>
> The current calculation for the number of Key Groups (max parallelism) leads
> in many cases to data skew and to confusion among users.
> Specifically, the fact that for maxParallelisms above 128, the default value
> is set to {{roundToPowerOfTwo(1.5 x parallelism)}} means that frequently,
> half of the tasks get one keygroup and the other half gets two keygroups,
> which is very skewed.
> See section (1) in this "lessons learned" blog post.
> https://engineering.contentsquare.com/2021/ten-flink-gotchas/
> We can fix this by
> - either setting a default maxParallelism to something pretty high (2048
> for example). The cost is that we add the default key group overhead per
> state entry from one byte to two bytes.
> - or we stay with some similar logic, but we instead of {{1.5 x
> operatorParallelism}} we go with some higher multiplier, like {{4 x
> operatorParallelism}}. The price is again that we more quickly reach the
> point where we have two bytes of keygroup encoding overhead, instead of one.
> Implementation wise, there is an unfortunate situation that the
> maxParallelism, if not configured, is not stored anywhere in the job graph,
> but re-derived on the JobManager each time it loads a JobGraph vertex
> (ExecutionJobVertex) which does not have a MaxParallelism configured. This
> relies on the implicit contract that this logic never changes.
> Changing this logic will instantly break all jobs which have not explicitly
> configured the Max Parallelism. That seems like a pretty heavy design
> shortcoming, unfortunately :-(
> A way to partially work around that is by moving the logic that derives the
> maximum parallelism to the {{StreamGraphGenerator}}, so we never create
> JobGraphs where vertices have no configured Max Parallelism (and we keep the
> re-derivation logic for backwards compatibility for persisted JobGraphs).
> The {{StreamExecutionEnvironment}} will need a flag to use the "old mode" to
> give existing un-configured applications a way to keep restoring from old
> savepoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21103) E2e tests time out on azure
[
https://issues.apache.org/jira/browse/FLINK-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-21103:
--
Component/s: (was: Runtime / State Backends)
Thanks for the follow up [~dwysakowicz] and [~rmetzger]!
I'm removing `State Backends` from the JIRA component field since there seems
to be no more clue on state backend related issues. Please feel free to add it
back if any new findings or I'm missing anything. Thanks.
> E2e tests time out on azure
> ---
>
> Key: FLINK-21103
> URL: https://issues.apache.org/jira/browse/FLINK-21103
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.1, 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12377=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> Creating worker2 ... done
> Jan 22 13:16:17 Waiting for hadoop cluster to come up. We have been trying
> for 0 seconds, retrying ...
> Jan 22 13:16:22 Waiting for hadoop cluster to come up. We have been trying
> for 5 seconds, retrying ...
> Jan 22 13:16:27 Waiting for hadoop cluster to come up. We have been trying
> for 10 seconds, retrying ...
> Jan 22 13:16:32 Waiting for hadoop cluster to come up. We have been trying
> for 15 seconds, retrying ...
> Jan 22 13:16:37 Waiting for hadoop cluster to come up. We have been trying
> for 20 seconds, retrying ...
> Jan 22 13:16:43 Waiting for hadoop cluster to come up. We have been trying
> for 26 seconds, retrying ...
> Jan 22 13:16:48 Waiting for hadoop cluster to come up. We have been trying
> for 31 seconds, retrying ...
> Jan 22 13:16:53 Waiting for hadoop cluster to come up. We have been trying
> for 36 seconds, retrying ...
> Jan 22 13:16:58 Waiting for hadoop cluster to come up. We have been trying
> for 41 seconds, retrying ...
> Jan 22 13:17:03 Waiting for hadoop cluster to come up. We have been trying
> for 46 seconds, retrying ...
> Jan 22 13:17:08 We only have 0 NodeManagers up. We have been trying for 0
> seconds, retrying ...
> 21/01/22 13:17:10 INFO client.RMProxy: Connecting to ResourceManager at
> master.docker-hadoop-cluster-network/172.19.0.3:8032
> 21/01/22 13:17:11 INFO client.AHSProxy: Connecting to Application History
> server at master.docker-hadoop-cluster-network/172.19.0.3:10200
> Jan 22 13:17:11 We now have 2 NodeManagers up.
> ==
> === WARNING: This E2E Run took already 80% of the allocated time budget of
> 250 minutes ===
> ==
> ==
> === WARNING: This E2E Run will time out in the next few minutes. Starting to
> upload the log output ===
> ==
> ##[error]The task has timed out.
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.0' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.1' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.2' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20496) RocksDB partitioned index filter option
[
https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li closed FLINK-20496.
-
Resolution: Implemented
Merged into master via:
be78727ae125c036cbb297d020e8a7ad23aae083
f7439740e8e023d458d2ac0cb2a58682eb9b6beb
Will add some release note and create new JIRA to add some instructions on when
and how to use this feature in our document later.
> RocksDB partitioned index filter option
> ---
>
> Key: FLINK-20496
> URL: https://issues.apache.org/jira/browse/FLINK-20496
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.10.2, 1.11.2, 1.12.0
>Reporter: YufeiLiu
>Assignee: YufeiLiu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> When using RocksDBStateBackend and enabling
> {{state.backend.rocksdb.memory.managed}} and
> {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited
> rocksdb memory usage which contains "write buffer" and "block cache". With
> these options rocksdb stores index and filters in block cache, because in
> default options index/filters can grows unlimited.
> But it's lead another issue, if high-priority cache(configure by
> {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all
> index/filters blocks, it will load all metadata from disk when cache missed,
> and program went extremely slow. According to [Partitioned Index
> Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1],
> we can enable two-level index having acceptable performance when
> index/filters cache missed.
> Enable these options can get over 10x faster in my case[2], I think we can
> add an option {{state.backend.rocksdb.partitioned-index-filters}} and default
> value is false, so we can use this feature easily.
> [1] Partitioned Index Filters:
> https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M,
> SSD, elapsed time 4.91ms -> 0.33ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21103) E2e tests time out on azure
[
https://issues.apache.org/jira/browse/FLINK-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293582#comment-17293582
]
Yu Li commented on FLINK-21103:
---
Checking the latest two building logs in the above comments, we could see the
below cases run for specially long time, and I suggest the PyFlink and SQL
component owners to take a look [~dianfu] [~jark]:
{noformat}
[PASS] 'Run kubernetes pyflink application test' passed after 12 minutes and 49
seconds!
[PASS] 'Running Kerberized YARN per-job on Docker test (default input)' passed
after 8 minutes and 24 seconds!
[PASS] 'TPC-DS end-to-end test (Blink planner)' passed after 19 minutes and 54
seconds!
[PASS] 'PyFlink end-to-end test' passed after 14 minutes and 41 seconds!
[PASS] 'PyFlink YARN per-job on Docker test' passed after 15 minutes and 26
seconds!
{noformat}
Logs I checked:
*
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13963=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
*
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13966=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> E2e tests time out on azure
> ---
>
> Key: FLINK-21103
> URL: https://issues.apache.org/jira/browse/FLINK-21103
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Runtime / State
> Backends, Tests
>Affects Versions: 1.12.1, 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12377=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> Creating worker2 ... done
> Jan 22 13:16:17 Waiting for hadoop cluster to come up. We have been trying
> for 0 seconds, retrying ...
> Jan 22 13:16:22 Waiting for hadoop cluster to come up. We have been trying
> for 5 seconds, retrying ...
> Jan 22 13:16:27 Waiting for hadoop cluster to come up. We have been trying
> for 10 seconds, retrying ...
> Jan 22 13:16:32 Waiting for hadoop cluster to come up. We have been trying
> for 15 seconds, retrying ...
> Jan 22 13:16:37 Waiting for hadoop cluster to come up. We have been trying
> for 20 seconds, retrying ...
> Jan 22 13:16:43 Waiting for hadoop cluster to come up. We have been trying
> for 26 seconds, retrying ...
> Jan 22 13:16:48 Waiting for hadoop cluster to come up. We have been trying
> for 31 seconds, retrying ...
> Jan 22 13:16:53 Waiting for hadoop cluster to come up. We have been trying
> for 36 seconds, retrying ...
> Jan 22 13:16:58 Waiting for hadoop cluster to come up. We have been trying
> for 41 seconds, retrying ...
> Jan 22 13:17:03 Waiting for hadoop cluster to come up. We have been trying
> for 46 seconds, retrying ...
> Jan 22 13:17:08 We only have 0 NodeManagers up. We have been trying for 0
> seconds, retrying ...
> 21/01/22 13:17:10 INFO client.RMProxy: Connecting to ResourceManager at
> master.docker-hadoop-cluster-network/172.19.0.3:8032
> 21/01/22 13:17:11 INFO client.AHSProxy: Connecting to Application History
> server at master.docker-hadoop-cluster-network/172.19.0.3:10200
> Jan 22 13:17:11 We now have 2 NodeManagers up.
> ==
> === WARNING: This E2E Run took already 80% of the allocated time budget of
> 250 minutes ===
> ==
> ==
> === WARNING: This E2E Run will time out in the next few minutes. Starting to
> upload the log output ===
> ==
> ##[error]The task has timed out.
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.0' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.1' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.2' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21515) SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is failing
[
https://issues.apache.org/jira/browse/FLINK-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-21515:
--
Fix Version/s: (was: 1.12.2)
1.12.3
Change fix version from 1.12.2 to 1.12.3 since the latest 1.12.2 release
candidate (rc2) is cut one commit (e9af362) before this one (b768a21). We can
change the fix version back if any new RC produced for 1.12.2
> SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is
> failing
>
>
> Key: FLINK-21515
> URL: https://issues.apache.org/jira/browse/FLINK-21515
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Matthias
>Assignee: Roman Khachatryan
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0, 1.12.3
>
>
> We experience a test instability with
> {{SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource}}.
> The test is occassionally timing out.
> See [this
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=290=logs=cc649950-03e9-5fae-8326-2f1ad744b536=51cab6ca-669f-5dc0-221d-1e4f7dc4fc85=7846]
> being related to FLINK-21030.
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f72fc00b800 nid=0x2133 runnable
> [0x7f73046ed000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskMailboxTestHarness.waitForTaskCompletion(StreamTaskMailboxTestHarness.java:147)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource(SourceStreamTaskTest.java:604)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
> This failure was reproducible on {{master}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend
[
https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned FLINK-21413:
-
Assignee: Jiayi Liao
Thanks [~wind_ljy], JIRA assigned.
> TtlMapState and TtlListState cannot be clean completely with Filesystem
> StateBackend
>
>
> Key: FLINK-21413
> URL: https://issues.apache.org/jira/browse/FLINK-21413
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Assignee: Jiayi Liao
>Priority: Major
> Attachments: image-2021-02-19-11-13-58-672.png
>
>
> Take the #TtlMapState as an example,
>
> {code:java}
> public Map> getUnexpiredOrNull(@Nonnull Map TtlValue> ttlValue) {
> Map> unexpired = new HashMap<>();
> TypeSerializer> valueSerializer =
> ((MapSerializer>)
> original.getValueSerializer()).getValueSerializer();
> for (Map.Entry> e : ttlValue.entrySet()) {
> if (!expired(e.getValue())) {
> // we have to do the defensive copy to update the
> value
> unexpired.put(e.getKey(),
> valueSerializer.copy(e.getValue()));
> }
> }
> return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
> }
> {code}
>
> The returned value will never be null and the #StateEntry will exists
> forever, which leads to memory leak if the key's range of the stream is very
> large. Below we can see that 20+ millison uncleared TtlStateMap could take up
> several GB memory.
>
> !image-2021-02-19-11-13-58-672.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend
[
https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288229#comment-17288229
]
Yu Li commented on FLINK-21413:
---
[~wind_ljy] would you like to work on the PR following the above solution? I
will assign this JIRA to you if so and help review the PR, or just feel free to
let me know if you're not available at this moment.
> TtlMapState and TtlListState cannot be clean completely with Filesystem
> StateBackend
>
>
> Key: FLINK-21413
> URL: https://issues.apache.org/jira/browse/FLINK-21413
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Priority: Major
> Attachments: image-2021-02-19-11-13-58-672.png
>
>
> Take the #TtlMapState as an example,
>
> {code:java}
> public Map> getUnexpiredOrNull(@Nonnull Map TtlValue> ttlValue) {
> Map> unexpired = new HashMap<>();
> TypeSerializer> valueSerializer =
> ((MapSerializer>)
> original.getValueSerializer()).getValueSerializer();
> for (Map.Entry> e : ttlValue.entrySet()) {
> if (!expired(e.getValue())) {
> // we have to do the defensive copy to update the
> value
> unexpired.put(e.getKey(),
> valueSerializer.copy(e.getValue()));
> }
> }
> return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
> }
> {code}
>
> The returned value will never be null and the #StateEntry will exists
> forever, which leads to memory leak if the key's range of the stream is very
> large. Below we can see that 20+ millison uncleared TtlStateMap could take up
> several GB memory.
>
> !image-2021-02-19-11-13-58-672.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend
[
https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287862#comment-17287862
]
Yu Li commented on FLINK-21413:
---
Checking the [state TTL FLIP
document|https://cwiki.apache.org/confluence/display/FLINK/FLIP-25%3A+Support+User+State+TTL+Natively#FLIP25:SupportUserStateTTLNatively-TTLbehaviour]
I cannot find description on whether TTL for a whole map is supported for
{{MapState}}, but according to the current implementation the answer is no (TTL
is only checked against value of each map entry). What's more, in
[HeapMapState#remove|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapMapState.java#L130-L132]
we could see the whole map will be removed if become empty, so I don't think
{{TtlIncrementalCleanup}} need to take care of the empty map case.
Accordingly, I think we should have a fast path in
{{TtlMapState#getUnexpiredOrNull}} to check whether {{ttlValue}} is empty and
return it directly (instead of returning {{NULL}}) if so, and returning
{{NULL}} iif {{ttlValue}} is not empty but all values expired
({{unexpired.size()}} is zero).
And similar logic should be applied to {{TtlListState#getUnexpiredOrNull}}.
Please let me know your thoughts [~wind_ljy]. Thanks.
> TtlMapState and TtlListState cannot be clean completely with Filesystem
> StateBackend
>
>
> Key: FLINK-21413
> URL: https://issues.apache.org/jira/browse/FLINK-21413
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Priority: Major
> Attachments: image-2021-02-19-11-13-58-672.png
>
>
> Take the #TtlMapState as an example,
>
> {code:java}
> public Map> getUnexpiredOrNull(@Nonnull Map TtlValue> ttlValue) {
> Map> unexpired = new HashMap<>();
> TypeSerializer> valueSerializer =
> ((MapSerializer>)
> original.getValueSerializer()).getValueSerializer();
> for (Map.Entry> e : ttlValue.entrySet()) {
> if (!expired(e.getValue())) {
> // we have to do the defensive copy to update the
> value
> unexpired.put(e.getKey(),
> valueSerializer.copy(e.getValue()));
> }
> }
> return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
> }
> {code}
>
> The returned value will never be null and the #StateEntry will exists
> forever, which leads to memory leak if the key's range of the stream is very
> large. Below we can see that 20+ millison uncleared TtlStateMap could take up
> several GB memory.
>
> !image-2021-02-19-11-13-58-672.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend
[
https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287550#comment-17287550
]
Yu Li commented on FLINK-21413:
---
Indeed, thanks for reporting the issue [~wind_ljy]. Would you like to work on
this and supply a PR to fix it? Thanks.
> TtlMapState and TtlListState cannot be clean completely with Filesystem
> StateBackend
>
>
> Key: FLINK-21413
> URL: https://issues.apache.org/jira/browse/FLINK-21413
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Priority: Major
> Attachments: image-2021-02-19-11-13-58-672.png
>
>
> Take the #TtlMapState as an example,
>
> {code:java}
> public Map> getUnexpiredOrNull(@Nonnull Map TtlValue> ttlValue) {
> Map> unexpired = new HashMap<>();
> TypeSerializer> valueSerializer =
> ((MapSerializer>)
> original.getValueSerializer()).getValueSerializer();
> for (Map.Entry> e : ttlValue.entrySet()) {
> if (!expired(e.getValue())) {
> // we have to do the defensive copy to update the
> value
> unexpired.put(e.getKey(),
> valueSerializer.copy(e.getValue()));
> }
> }
> return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
> }
> {code}
>
> The returned value will never be null and the #StateEntry will exists
> forever, which leads to memory leak if the key's range of the stream is very
> large. Below we can see that 20+ millison uncleared TtlStateMap could take up
> several GB memory.
>
> !image-2021-02-19-11-13-58-672.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16444) Count the read/write/seek/next latency of RocksDB as metrics
[
https://issues.apache.org/jira/browse/FLINK-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286910#comment-17286910
]
Yu Li commented on FLINK-16444:
---
After some more investigation, the metrics we plan to add here should compare
with [Kafka state store
metrics|https://kafka.apache.org/documentation/#kafka_streams_store_monitoring]
instead of [statistics and properties based
metrics|https://kafka.apache.org/documentation/#kafka_streams_rocksdb_monitoring].
I did some comparison between properties-based metrics listed in KIP-607 and
[those exist in
Flink|https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#rocksdb-native-metrics],
and confirmed that {{live-sst-files-size}} is the only one we're missing and
may consider to add.
Regarding the [statistics-based
metrics|https://github.com/facebook/rocksdb/wiki/Statistics], it seems we don't
have them in Flink. It seems to be a little bit off-track to discuss here and
let's create some other JIRAs to follow up.
> Count the read/write/seek/next latency of RocksDB as metrics
>
>
> Key: FLINK-16444
> URL: https://issues.apache.org/jira/browse/FLINK-16444
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, user cannot know the read/write/seek/next latency of RocksDB, we
> could add these helpful metrics to know the overall state performance. To not
> affect the action performance much, we could introduce counter to only record
> the latency at interval of some actions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16444) Count the read/write/seek/next latency of RocksDB as metrics
[ https://issues.apache.org/jira/browse/FLINK-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286615#comment-17286615 ] Yu Li commented on FLINK-16444: --- [~yunta] Besides these metrics, let's also check and confirm whether more metrics worth being introduced referring to [KIP-607|https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB]. > Count the read/write/seek/next latency of RocksDB as metrics > > > Key: FLINK-16444 > URL: https://issues.apache.org/jira/browse/FLINK-16444 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Fix For: 1.13.0 > > > Currently, user cannot know the read/write/seek/next latency of RocksDB, we > could add these helpful metrics to know the overall state performance. To not > affect the action performance much, we could introduce counter to only record > the latency at interval of some actions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20496) RocksDB partitioned index filter option
[
https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned FLINK-20496:
-
Fix Version/s: 1.13.0
Affects Version/s: 1.12.0
1.10.2
1.11.2
Assignee: YufeiLiu
Thanks for filing the JIRA [~liuyufei] and the proposal LGTM. I've assigned the
issue to you and will wait for the PR. Thanks.
btw, there're already some settings for caching index and filters in
[RocksDBResourceContainer|https://github.com/apache/flink/blob/59ae84069313ede60cf7ad3a9d2fe1bc07c4e460/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBResourceContainer.java#L147-L149]
when using managed memory for RocksDB backend, just in case you didn't notice.
> RocksDB partitioned index filter option
> ---
>
> Key: FLINK-20496
> URL: https://issues.apache.org/jira/browse/FLINK-20496
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.10.2, 1.12.0, 1.11.2
>Reporter: YufeiLiu
>Assignee: YufeiLiu
>Priority: Major
> Fix For: 1.13.0
>
>
> When using RocksDBStateBackend and enabling
> {{state.backend.rocksdb.memory.managed}} and
> {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited
> rocksdb memory usage which contains "write buffer" and "block cache". With
> these options rocksdb stores index and filters in block cache, because in
> default options index/filters can grows unlimited.
> But it's lead another issue, if high-priority cache(configure by
> {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all
> index/filters blocks, it will load all metadata from disk when cache missed,
> and program went extremely slow. According to [Partitioned Index
> Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1],
> we can enable two-level index having acceptable performance when
> index/filters cache missed.
> Enable these options can get over 10x faster in my case[2], I think we can
> add an option {{state.backend.rocksdb.partitioned-index-filters}} and default
> value is false, so we can use this feature easily.
> [1] Partitioned Index Filters:
> https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M,
> SSD, elapsed time 4.91ms -> 0.33ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-20287) Add documentation of how to switch memory allocator in Flink docker image
[ https://issues.apache.org/jira/browse/FLINK-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li resolved FLINK-20287. --- Resolution: Done Merged into: master via b457225208f41f593a1a795e988be21b646c55b6 release-1.12 via e1c0220984a91091618ccb653cee7b007077a2eb > Add documentation of how to switch memory allocator in Flink docker image > - > > Key: FLINK-20287 > URL: https://issues.apache.org/jira/browse/FLINK-20287 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Documentation >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > > Add documentation to tell user how to switch memory allocator in Flink docker > image. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19172) [AbstractFileStateBackend]
[
https://issues.apache.org/jira/browse/FLINK-19172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li closed FLINK-19172.
-
Resolution: Not A Problem
In Flink we use the code convention that method parameters are non-null unless
marked with {{@Nullable}} annotation, which also applies to the
{{AbstractFileStateBackend#validatePath}} method. And we could see the
[caller|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/AbstractFileStateBackend.java#L109-L110]
of this method follows this rule.
Please feel free to reopen the issue if you have any other concerns
[~alessiosavi]. Thanks.
> [AbstractFileStateBackend]
> --
>
> Key: FLINK-19172
> URL: https://issues.apache.org/jira/browse/FLINK-19172
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Runtime / State Backends
>Affects Versions: 1.8.0
>Reporter: Alessio Savi
>Priority: Minor
> Attachments: Flink.PNG
>
>
> The method `validatePath` of class `AbstractFileStateBackend` does not check
> if the pathPart retrived from the input `Path` is blank. Instead, it only
> check if it is null.
> Is this a bug?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-20288) Correct documentation about savepoint self-contained
[ https://issues.apache.org/jira/browse/FLINK-20288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li resolved FLINK-20288. --- Resolution: Duplicate [~klion26] thanks for the reminder, closing this one as duplication of FLINK-19381 and let's track the work there [~yunta]. > Correct documentation about savepoint self-contained > > > Key: FLINK-20288 > URL: https://issues.apache.org/jira/browse/FLINK-20288 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / Checkpointing >Affects Versions: 1.11.0 >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Savepoint self-contained has been supported while the documentation still > remain as not supported, we should fix that description. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19125) Avoid memory fragmentation when running flink docker image
[
https://issues.apache.org/jira/browse/FLINK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-19125:
--
Release Note: After FLINK-19125 Jemalloc is adopted as the default memory
allocator in flink docker image to prevent memory fragmentation problem, and
users could roll back to use glibc by passing the 'disable-jemalloc' flag to
the docker-entrypoint.sh script. More details please refer to flink
documentation. (was: Adopt Jemalloc as default memory allocator in official
docker image's docker-entrypoint.sh to avoid known memory fragmentation
problem, and user could also revert back to previous glibc if parameter
'disable-jemalloc' is given.)
> Avoid memory fragmentation when running flink docker image
> --
>
> Key: FLINK-19125
> URL: https://issues.apache.org/jira/browse/FLINK-19125
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Runtime / State Backends
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> This ticket tracks the problem of memory fragmentation when launching default
> Flink docker image.
> In FLINK-18712, user reported if he submits job with rocksDB state backend on
> a k8s session cluster again and again once it finished, the memory usage of
> task manager grows continuously until OOM killed.
> I reproduce this problem with official Flink docker image no matter how we
> use rocksDB (whether to enable managed memory or not).
> I dig into the problem and found this is due to the memory fragmentation
> caused by {{glibc}}, which would not return memory to kernel gracefully
> (please refer to [glibc
> bugzilla|https://sourceware.org/bugzilla/show_bug.cgi?id=15321] and [glibc
> manual|https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc])
> I found limiting MALLOC_ARENA_MAX to 2 could mitigate this problem (please
> refer to
> [choose-for-malloc_arena_max|https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior#what-value-to-choose-for-malloc_arena_max]
> for more details).
> And if we choose to use jemalloc to allocate memory via rebuilding another
> docker image, the problem would be gone.
> {code:java}
> apt-get -y install libjemalloc-dev
> ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so
> {code}
> Jemalloc intends to [emphasize fragmentation
> avoidance|https://github.com/jemalloc/jemalloc/wiki/Background#intended-use]
> and we might consider to re-factor our Dockerfile to base on jemalloc to
> avoid memory fragmentation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236881#comment-17236881 ] Yu Li commented on FLINK-15507: --- I second that. Let's revive the discussion and get this done in 1.13 after reaching a consensus. > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Assignee: Zakelly Lan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.13.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19125) Avoid memory fragmentation when running flink docker image
[
https://issues.apache.org/jira/browse/FLINK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236880#comment-17236880
]
Yu Li commented on FLINK-19125:
---
Thanks all for the efforts!
As mentioned in [ML
discussion|https://s.apache.org/release-note-for-changing-default-memory-allocator],
[~yunta] could you supplement the {{ReleaseNote}} here? Thanks.
> Avoid memory fragmentation when running flink docker image
> --
>
> Key: FLINK-19125
> URL: https://issues.apache.org/jira/browse/FLINK-19125
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Runtime / State Backends
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> This ticket tracks the problem of memory fragmentation when launching default
> Flink docker image.
> In FLINK-18712, user reported if he submits job with rocksDB state backend on
> a k8s session cluster again and again once it finished, the memory usage of
> task manager grows continuously until OOM killed.
> I reproduce this problem with official Flink docker image no matter how we
> use rocksDB (whether to enable managed memory or not).
> I dig into the problem and found this is due to the memory fragmentation
> caused by {{glibc}}, which would not return memory to kernel gracefully
> (please refer to [glibc
> bugzilla|https://sourceware.org/bugzilla/show_bug.cgi?id=15321] and [glibc
> manual|https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc])
> I found limiting MALLOC_ARENA_MAX to 2 could mitigate this problem (please
> refer to
> [choose-for-malloc_arena_max|https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior#what-value-to-choose-for-malloc_arena_max]
> for more details).
> And if we choose to use jemalloc to allocate memory via rebuilding another
> docker image, the problem would be gone.
> {code:java}
> apt-get -y install libjemalloc-dev
> ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so
> {code}
> Jemalloc intends to [emphasize fragmentation
> avoidance|https://github.com/jemalloc/jemalloc/wiki/Background#intended-use]
> and we might consider to re-factor our Dockerfile to base on jemalloc to
> avoid memory fragmentation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15747) Enable setting RocksDB log level from configuration
[
https://issues.apache.org/jira/browse/FLINK-15747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-15747:
--
Fix Version/s: (was: 1.12.0)
1.13.0
> Enable setting RocksDB log level from configuration
> ---
>
> Key: FLINK-15747
> URL: https://issues.apache.org/jira/browse/FLINK-15747
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Congxian Qiu
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently to open the RocksDB local log, one has to create a customized
> {{OptionsFactory}}, which is not quite convenient. This JIRA proposes to
> enable setting it from configuration in flink-conf.yaml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20269) The flush happens too frequent in SavepoinV2Serializer
[
https://issues.apache.org/jira/browse/FLINK-20269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236867#comment-17236867
]
Yu Li commented on FLINK-20269:
---
[~wind_ljy] Please check and confirm whether this is a duplication of
FLINK-19325, thanks.
> The flush happens too frequent in SavepoinV2Serializer
> --
>
> Key: FLINK-20269
> URL: https://issues.apache.org/jira/browse/FLINK-20269
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.0
>Reporter: Jiayi Liao
>Priority: Major
>
> The reason I notice this is, I find that the metadata's persistence can be
> very slow (but the states' uploading process works fine) when the network is
> unstable, and almost every time I dump the stack of the process, the
> bottleneck happens on Hdfs client waiting for Datanodes' ack during
> metadata's persistance.
>
> I wonder, is it really necessary to flush the stream after every
> {{StreamStateHandle}}'s serialization?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-14482: -- Description: This JIRA aims at rebasing frocksdb to [newer version|https://github.com/facebook/rocksdb/releases] of official RocksDB. (was: Current rocksDB-5.17.2 does not support write buffer manager well, we need to bump rocksdb version to support that feature.) The description was written last year and needs some update. It's true that we have done necessary backports via FLINK-14483 to enable frocksdb to use write buffer manager, and this JIRA is kept open simply to rebase frocksdb to newer version of rocksdb. > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15747) Enable setting RocksDB log level from configuration
[
https://issues.apache.org/jira/browse/FLINK-15747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229002#comment-17229002
]
Yu Li commented on FLINK-15747:
---
[~qinjunjerry] I'm afraid not, checking the
[PR|https://github.com/ververica/frocksdb/pull/12] we could see there're some
unresolved problem, and the work to upgrade frocksdb to higher version
(FLINK-14482) is also postponed due to performance regression, plus the fact
that 1.12.0 feature freeze date has passed, we probably need to postpone this
one to later release.
> Enable setting RocksDB log level from configuration
> ---
>
> Key: FLINK-15747
> URL: https://issues.apache.org/jira/browse/FLINK-15747
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Congxian Qiu
>Priority: Major
> Fix For: 1.12.0
>
>
> Currently to open the RocksDB local log, one has to create a customized
> {{OptionsFactory}}, which is not quite convenient. This JIRA proposes to
> enable setting it from configuration in flink-conf.yaml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228997#comment-17228997 ] Yu Li commented on FLINK-14482: --- Some update here: [~yunta] and I have verified that 5.18.3 with the fix of FLINK-19710 could get the same performance as 5.17.2, but still observed ~5% regression on the list/map get/iterator benchmarks with 6.x, indicating there're new issues introduced in the higher versions, so we decide to postpone the upgrade to locate the new issues. We aim at completing the upgrade to 6.x in 1.13.0 release. > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Current rocksDB-5.17.2 does not support write buffer manager well, we need to > bump rocksdb version to support that feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
