[GitHub] flink pull request #6174: [FLINNK-9601][state]Try to reuse the snapshotData ...

2018-06-16 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6174 [FLINNK-9601][state]Try to reuse the snapshotData array as the partitioned destination on… ## What is the purpose of the change In `CopyOnWriteStateTableSnapshot`, we only reuse

[GitHub] flink issue #6159: [FLINK-9487] Prepare InternalTimerHeap for asynchronous s...

2018-06-16 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6159 @StefanRRichter Sorry that I didn't take a look again yesterday, I was a bit too busy at that time. I had a look at the fix for the `CopyOnWriteStateTable`, I afraid it's still a partial fix, I

[GitHub] flink pull request #6164: [FLINK-9584][connector] Properly close the output ...

2018-06-14 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6164 [FLINK-9584][connector] Properly close the output stream in BucketingSink & RollingSink. ## What is the purpose of the change This PR fixes the unclosed streams in Bucketing-/Rolling

[GitHub] flink pull request #6159: [FLINK-9487] Prepare InternalTimerHeap for asynchr...

2018-06-13 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6159#discussion_r195299958 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractKeyGroupPartitioner.java --- @@ -0,0 +1,227 @@ +/* + * Licensed

[GitHub] flink pull request #6159: [FLINK-9487] Prepare InternalTimerHeap for asynchr...

2018-06-13 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6159#discussion_r195156251 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractKeyGroupPartitioner.java --- @@ -0,0 +1,227 @@ +/* + * Licensed

[GitHub] flink pull request #6159: [FLINK-9487] Prepare InternalTimerHeap for asynchr...

2018-06-13 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6159#discussion_r195168815 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -224,6 +227,15 @@ void

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-13 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r195019971 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBListState.java --- @@ -237,4

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194952224 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBAggregatingState.java

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194951927 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBAggregatingState.java

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194951490 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/AbstractRocksDBAppendingState.java

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194951510 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/AbstractHeapAppendingState.java --- @@ -0,0 +1,48 @@ +package

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194951373 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/AbstractRocksDBAppendingState.java

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194947600 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/AbstractRocksDBAppendingState.java

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194946663 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBAggregatingState.java

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194795632 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBListState.java --- @@ -237,4

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194796355 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBListState.java --- @@ -94,6

[GitHub] flink pull request #6156: [FLINK-9572] Extend InternalAppendingState with in...

2018-06-12 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6156#discussion_r194790994 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBFoldingState.java --- @@ -87,40

[GitHub] flink pull request #6144: [hotfix][javadoc]Fix the incorrect java doc for Op...

2018-06-09 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6144 [hotfix][javadoc]Fix the incorrect java doc for OptionsFactory. ## What is the purpose of the change *This PR simply fixes the incorrect java doc of `OptionsFactory`.* ## Brief

[GitHub] flink pull request #6135: [FLINK-9546][core] Fix the checking of heartbeatTi...

2018-06-07 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6135 [FLINK-9546][core] Fix the checking of heartbeatTimeoutIntervalMs in HeartbeatMonitor. ## What is the purpose of the change The heartbeatTimeoutIntervalMs of HeartbeatMonitor should

[GitHub] flink issue #6135: [FLINK-9546][core] Fix the checking of heartbeatTimeoutIn...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6135 cc @tillrohrmann ---

[GitHub] flink issue #6132: [FLINK-9456][Distributed Coordination]Let ResourceManager...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6132 the failure on travis is unrelated. ---

[GitHub] flink pull request #6130: [FLINK-9545] Support read a file multiple times in...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6130#discussion_r193638907 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java --- @@ -1355,26 +1541,39

[GitHub] flink pull request #6130: [FLINK-9545] Support read a file multiple times in...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6130#discussion_r193638176 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java --- @@ -1161,7 +1301,53 @@ public

[GitHub] flink pull request #6130: [FLINK-9545] Support read a file multiple times in...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6130#discussion_r193637932 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java --- @@ -1161,7 +1301,53 @@ public

[GitHub] flink pull request #6130: [FLINK-9545] Support read a file multiple times in...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6130#discussion_r193637758 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java --- @@ -1093,6 +1178,59 @@ public

[GitHub] flink pull request #6130: [FLINK-9545] Support read a file multiple times in...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6130#discussion_r193637243 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java --- @@ -970,6 +991,37 @@ public

[GitHub] flink issue #6133: [FLINK-9351][Distributed Coordination] RM stop assigning ...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6133 cc @tillrohrmann ---

[GitHub] flink issue #6132: [FLINK-9456][Distributed Coordination]Let ResourceManager...

2018-06-07 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6132 cc @tillrohrmann ---

[GitHub] flink pull request #6133: [FLINK-9351][Distributed Coordination] RM stop ass...

2018-06-07 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6133 [FLINK-9351][Distributed Coordination] RM stop assigning slot to Job because the TM killed before connecting to JM successfully ## What is the purpose of the change *This PR bases

[GitHub] flink pull request #6132: [FLINK-9456][Distributed Coordination]Let Resource...

2018-06-06 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6132 [FLINK-9456][Distributed Coordination]Let ResourceManager notify JobManager about failed/killed TaskManagers. ## What is the purpose of the change *Often, the ResourceManager learns

[GitHub] flink issue #5582: [FLINK-8790][State] Improve performance for recovery from...

2018-06-05 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5582 Hi @StefanRRichter I updated the PR according to the previous discussions, could you please have a look when you have time? The travis failed is unrelated, it's a checkstyle error introduced

[GitHub] flink issue #6117: [hotfix][checkstyle] fix checkstyle of CEPOperatorTest

2018-06-05 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6117 cc @zentol ---

[GitHub] flink pull request #6117: [hotfix][checkstyle] fix checkstyle of CEPOperator...

2018-06-05 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6117 [hotfix][checkstyle] fix checkstyle of CEPOperatorTest ## What is the purpose of the change *fixes checkstyle error introduced by the previous PR.* ## Brief change log

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-06-04 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r192732891 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -406,6 +407,43 @@ public void onError(Throwable error

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-06-04 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r192733056 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -406,6 +407,43 @@ public void onError(Throwable error

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-06-04 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r192733087 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -406,6 +407,43 @@ public void onError(Throwable error

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-06-04 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r192723013 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -356,7 +363,8 @@ public void onContainersAllocated(List

[GitHub] flink pull request #4949: [FLINK-7866] [runtime] Weigh list of preferred loc...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/4949#discussion_r192407287 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/SlotProfile.java --- @@ -203,68 +206,104 @@ public

[GitHub] flink pull request #4949: [FLINK-7866] [runtime] Weigh list of preferred loc...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/4949#discussion_r192406775 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/SlotProfile.java --- @@ -203,68 +206,104 @@ public

[GitHub] flink pull request #4949: [FLINK-7866] [runtime] Weigh list of preferred loc...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/4949#discussion_r192402396 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/SlotProfile.java --- @@ -203,68 +206,104 @@ public

[GitHub] flink pull request #4949: [FLINK-7866] [runtime] Weigh list of preferred loc...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/4949#discussion_r192384844 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/SlotProfile.java --- @@ -203,68 +206,104 @@ public

[GitHub] flink pull request #5982: [FLINK-9325][checkpoint]generate the meta file for...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5982#discussion_r192374992 --- Diff: flink-core/src/main/java/org/apache/flink/core/fs/ClosingAtomicCreatingFSDataOutputStream.java --- @@ -0,0 +1,129 @@ +/* + * Licensed

[GitHub] flink issue #5582: [FLINK-8790][State] Improve performance for recovery from...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5582 @StefanRRichter Thanks for your nice review and preventing this PR to fall into a sick way, I will change the code according to your comments and ping you again when I finish this. ---

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192352087 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192350746 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192348975 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192344507 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192342710 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192341305 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192336298 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192335886 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192335797 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192332556 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-06-01 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192331980 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink issue #5582: [FLINK-8790][State] Improve performance for recovery from...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5582 @StefanRRichter Thanks for your nice review, addressed your comments, could you please have a look again? ---

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192082241 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192081415 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192081137 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192080706 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192078870 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192078846 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtilsTest.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192078816 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192076835 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192075842 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192075543 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192075519 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192065096 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeySerializationUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-31 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r192026051 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeySerializationUtils.java

[GitHub] flink issue #5879: [FLINK-9215][resoucemanager] Reduce noise in SlotPool's l...

2018-05-30 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5879 @dawidwys It's ok with me, the unrelated change should be a minor java doc hot fix. ---

[GitHub] flink issue #5982: [FLINK-9325][checkpoint]generate the meta file for checkp...

2018-05-29 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5982 @StephanEwen I've addressed your comments, could you please have a look again? ---

[GitHub] flink issue #6094: [FLINK-9468][filesystem] fix calculate outputLimit incorr...

2018-05-29 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6094 CC @StephanEwen ---

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-29 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r191453657 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeySerializationUtils.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-29 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r191447222 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #5582: [FLINK-8790][State] Improve performance for recove...

2018-05-29 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/5582#discussion_r191446989 --- Diff: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeySerializationUtils.java

[GitHub] flink pull request #6094: [FLINK-9468][filesystem] fix get outputLimit in Li...

2018-05-29 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6094 [FLINK-9468][filesystem] fix get outputLimit in LimitedConnectionsFileSystem#createStream() incorrectly ## What is the purpose of the change This PR fixes the bug of getting `outputLimit

[GitHub] flink issue #5982: [FLINK-9325][checkpoint]generate the meta file for checkp...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5982 Yes, @StephanEwen thanks for the continuous suggestions, will follow your suggestion. ---

[GitHub] flink issue #6077: [FLINK-9436][state] Remove generic parameter namespace fr...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6077 +1 ---

[GitHub] flink issue #5582: [FLINK-8790][State] Improve performance for recovery from...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5582 Hi @StefanRRichter could you please have a look at this? ---

[GitHub] flink pull request #6088: [FLINK-9417][ Distributed Coordination] Send heart...

2018-05-28 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6088 [FLINK-9417][ Distributed Coordination] Send heartbeat requests from RPC endpoint's main thread ## What is the purpose of the change This PR try to send heartbeat requests from RPC

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r191162993 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -212,13 +214,12 @@ private void getContainersFromPreviousAttempts

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r191162395 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -510,4 +509,39 @@ private int generatePriority(ResourceProfile

[GitHub] flink pull request #6087: [FLINK-9410] [yarn] Replace NMClient with NMClient...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6087#discussion_r191162507 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java --- @@ -510,4 +509,39 @@ private int generatePriority(ResourceProfile

[GitHub] flink issue #6087: [FLINK-9410] [yarn] Replace NMClient with NMClientAsync i...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6087 Hi @zhangminglei , I would suggest to retry, this is more consistent with the behavior when we are using the sync API, in fact, currently if we don't retry to start container then the job

[GitHub] flink issue #6087: [FLINK-9410] [yarn] Replace NMClient with NMClientAsync i...

2018-05-28 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6087 Hi @zhangminglei , do you want to take over this? if yes, plz free feel to take over this. I had a brief look at the code, I think this may not as easy as replace all api() to apiAsync(), e.g

[GitHub] flink issue #6062: [FLINK-9423][state] Implement efficient deletes for heap-...

2018-05-25 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6062 @StefanRRichter Thanks for your reply, I think I should also have a look at https://github.com/apache/flink/pull/3359 to supplement my knowledge on this. Looking forward your design document! ---

[GitHub] flink issue #6062: [FLINK-9423][state] Implement efficient deletes for heap-...

2018-05-25 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6062 Hi @StefanRRichter could I ask one more thing that I'm very interested in? From this PR I can feel that the `InternalTimerHeap` is really like a `HeapState` which scoped to the key group(not per

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-24 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190517094 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -458,24 +458,33 @@ private int

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-24 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190517336 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimer.java --- @@ -23,8 +23,8 @@ /** * Internal

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190463723 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/TimerHeapInternalTimer.java --- @@ -0,0 +1,235

[GitHub] flink issue #5582: [FLINK-8790][State] Improve performance for recovery from...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5582 Unfortunately, after confirming with RocksDB, the `deleteRange()` is still an experimental feature, it may have impact on read performance currently(event thought we could use the ReadOption

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190311650 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190295347 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190294307 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190293256 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190290965 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190288320 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190285625 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/TimerHeapInternalTimer.java --- @@ -0,0 +1,235

[GitHub] flink pull request #6062: [FLINK-9423][state] Implement efficient deletes fo...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on a diff in the pull request: https://github.com/apache/flink/pull/6062#discussion_r190279440 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerHeap.java --- @@ -0,0 +1,504

[GitHub] flink issue #6063: [FLINK-9426][test] Harden RocksDBWriteBatchPerformanceTes...

2018-05-23 Thread sihuazhou
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/6063 cc @StefanRRichter I'm sorry for introduce this unstable tests... ---

[GitHub] flink pull request #6063: [FLINK-9426][test] Harden RocksDBWriteBatchPerform...

2018-05-23 Thread sihuazhou
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/6063 [FLINK-9426][test] Harden RocksDBWriteBatchPerformanceTest.benchMark() ## What is the purpose of the change We use the assert to check the performance of WriteBatch is better than Put

<    1   2   3   4   5   6   >