[jira] [Created] (FLINK-8086) FlinkKafkaProducer011 can permanently fail in recovery

2017-11-15 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-8086: - Summary: FlinkKafkaProducer011 can permanently fail in recovery Key: FLINK-8086 URL: https://issues.apache.org/jira/browse/FLINK-8086 Project: Flink Issue

[jira] [Updated] (FLINK-8086) FlinkKafkaProducer011 can permanently fail in recovery through ProducerFencedException

2017-11-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-8086: -- Summary: FlinkKafkaProducer011 can permanently fail in recovery through ProducerFencedException

[jira] [Updated] (FLINK-8086) FlinkKafkaProducer011 can permanently fail in recovery through ProducerFencedException

2017-11-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-8086: -- Description: Chaos monkey test in a cluster environment can permanently bring down our

[jira] [Commented] (FLINK-7880) flink-queryable-state-java fails with core-dump

2017-11-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239068#comment-16239068 ] Stefan Richter commented on FLINK-7880: --- My theory is that the `dispose()` is not properly executed

[jira] [Commented] (FLINK-7880) flink-queryable-state-java fails with core-dump

2017-11-06 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240153#comment-16240153 ] Stefan Richter commented on FLINK-7880: --- Yes, but the test seems to expect that waiting for

[jira] [Comment Edited] (FLINK-7880) flink-queryable-state-java fails with core-dump

2017-11-06 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240153#comment-16240153 ] Stefan Richter edited comment on FLINK-7880 at 11/6/17 11:20 AM: - Yes, but

[jira] [Commented] (FLINK-7873) Introduce CheckpointCacheManager for reading checkpoint data locally when performing failover

2017-10-31 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233608#comment-16233608 ] Stefan Richter commented on FLINK-7873: --- The first proposal of this JIRA had some issues. The

[jira] [Comment Edited] (FLINK-7873) Introduce CheckpointCacheManager for reading checkpoint data locally when performing failover

2017-10-31 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233608#comment-16233608 ] Stefan Richter edited comment on FLINK-7873 at 11/1/17 3:36 AM: The first

[jira] [Created] (FLINK-9304) Timer service shutdown should not be interrupted

2018-05-07 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9304: - Summary: Timer service shutdown should not be interrupted Key: FLINK-9304 URL: https://issues.apache.org/jira/browse/FLINK-9304 Project: Flink Issue Type:

[jira] [Commented] (FLINK-9302) Checkpoints continues to fail when using filesystem state backend with CIRCULAR REFERENCE:java.io.IOException

2018-05-07 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465575#comment-16465575 ] Stefan Richter commented on FLINK-9302: --- If you take a look at this stack trace, it tells you that

[jira] [Closed] (FLINK-9302) Checkpoints continues to fail when using filesystem state backend with CIRCULAR REFERENCE:java.io.IOException

2018-05-07 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9302. - Resolution: Not A Problem > Checkpoints continues to fail when using filesystem state backend

[jira] [Closed] (FLINK-9269) Concurrency problem in HeapKeyedStateBackend when performing checkpoint async

2018-05-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9269. - Resolution: Fixed Merged in: master: 14e7d35f26 release-1.5: 3ba21adc0e > Concurrency problem

[jira] [Created] (FLINK-9355) Simplify configuration of local recovery to a simple on/off

2018-05-14 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9355: - Summary: Simplify configuration of local recovery to a simple on/off Key: FLINK-9355 URL: https://issues.apache.org/jira/browse/FLINK-9355 Project: Flink

[jira] [Commented] (FLINK-9302) Checkpoints continues to fail when using filesystem state backend with CIRCULAR REFERENCE:java.io.IOException

2018-05-08 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467013#comment-16467013 ] Stefan Richter commented on FLINK-9302: --- Thanks for the update, I appreciate it. This information

[jira] [Closed] (FLINK-8992) Implement source and operator that validate exactly-once

2018-04-27 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8992. - Resolution: Fixed Merged in: master: 31c717697a52af2c64439aaaba2f6a1f97a22159 release 1.5: 

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-04-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457505#comment-16457505 ] Stefan Richter commented on FLINK-9268: --- This is a known issue with RocksDB, see

[jira] [Closed] (FLINK-9254) Move NotSoMiniClusterIterations to be an end-to-end test

2018-05-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9254. - Resolution: Fixed Merged in: master: b50cebb656 release-1.5: ed3447e343 > Move

[jira] [Reopened] (FLINK-8978) End-to-end test: Job upgrade

2018-05-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reopened FLINK-8978: --- Change fix version > End-to-end test: Job upgrade > > >

[jira] [Closed] (FLINK-8978) End-to-end test: Job upgrade

2018-05-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8978. - Resolution: Fixed Merged in: master: 5ac4d29609 release-1.5: 54befe5a31 > End-to-end test: Job

[jira] [Updated] (FLINK-8978) End-to-end test: Job upgrade

2018-05-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-8978: -- Fix Version/s: (was: 1.5.1) 1.5.0 > End-to-end test: Job upgrade >

[jira] [Closed] (FLINK-8978) End-to-end test: Job upgrade

2018-05-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8978. - Resolution: Fixed > End-to-end test: Job upgrade > > >

[jira] [Created] (FLINK-9388) Inconsistency in job shutdown produces confusing log message

2018-05-17 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9388: - Summary: Inconsistency in job shutdown produces confusing log message Key: FLINK-9388 URL: https://issues.apache.org/jira/browse/FLINK-9388 Project: Flink

[jira] [Commented] (FLINK-9373) Fix potential data losing for RocksDBBackend

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478861#comment-16478861 ] Stefan Richter commented on FLINK-9373: --- [~sihuazhou] Will you be able to do this quickly or can I

[jira] [Commented] (FLINK-9373) Fix potential data losing for RocksDBBackend

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478842#comment-16478842 ] Stefan Richter commented on FLINK-9373: --- Maybe we can also take the PR basically "as is" for now so

[jira] [Created] (FLINK-9390) Shutdown of KafkaProducer causes confusing log message

2018-05-17 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9390: - Summary: Shutdown of KafkaProducer causes confusing log message Key: FLINK-9390 URL: https://issues.apache.org/jira/browse/FLINK-9390 Project: Flink Issue

[jira] [Updated] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-8910: -- Fix Version/s: 1.5.0 > Introduce automated end-to-end test for local recovery (including sticky

[jira] [Closed] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8910. - Resolution: Fixed Release Note: We changed the default SLOT_IDLE_TIMEOUT to the

[jira] [Reopened] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reopened FLINK-8910: --- Change fix version > Introduce automated end-to-end test for local recovery (including sticky >

[jira] [Closed] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8910. - Resolution: Fixed > Introduce automated end-to-end test for local recovery (including sticky >

[jira] [Commented] (FLINK-9373) Fix potential data losing for RocksDBBackend

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478857#comment-16478857 ] Stefan Richter commented on FLINK-9373: --- Well, but we are also at risk to have only a partial fix.

[jira] [Commented] (FLINK-9373) Fix potential data losing for RocksDBBackend

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478866#comment-16478866 ] Stefan Richter commented on FLINK-9373: --- Great, thanks a lot! > Fix potential data losing for

[jira] [Created] (FLINK-9375) Introduce AbortCheckpoint message from JM to TMs

2018-05-16 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9375: - Summary: Introduce AbortCheckpoint message from JM to TMs Key: FLINK-9375 URL: https://issues.apache.org/jira/browse/FLINK-9375 Project: Flink Issue Type:

[jira] [Commented] (FLINK-9390) Shutdown of KafkaProducer causes confusing log message

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478936#comment-16478936 ] Stefan Richter commented on FLINK-9390: --- [~triones] Unfortunately I don't know a way to reproduce

[jira] [Closed] (FLINK-9373) Fix potential data losing for RocksDBBackend

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9373. - Resolution: Fixed Fix Version/s: 1.6.0 Merged in: master: 105b30686f release-1.5:

[jira] [Commented] (FLINK-9390) Shutdown of KafkaProducer causes confusing log message

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478941#comment-16478941 ] Stefan Richter commented on FLINK-9390: --- >From the log, it seems that Kafka09 connectors are used.

[jira] [Commented] (FLINK-9375) Introduce AbortCheckpoint message from JM to TMs

2018-05-17 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479298#comment-16479298 ] Stefan Richter commented on FLINK-9375: --- [~yanghua] this task maybe a bit more tricky than it

[jira] [Closed] (FLINK-8845) Use WriteBatch to improve performance for recovery in RocksDB backend

2018-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8845. - Resolution: Fixed Fix Version/s: 1.5.1 Merged in: master: 1c7341ad1a release-1.5: 

[jira] [Assigned] (FLINK-9423) Implement efficient deletes for heap based timer service

2018-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-9423: - Assignee: Stefan Richter > Implement efficient deletes for heap based timer service >

[jira] [Updated] (FLINK-9423) Implement efficient deletes for heap based timer service

2018-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-9423: -- Description: The current data structures in the `HeapInternalTimerService` are not able to

[jira] [Created] (FLINK-9423) Implement efficient deletes for heap based timer service

2018-05-23 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9423: - Summary: Implement efficient deletes for heap based timer service Key: FLINK-9423 URL: https://issues.apache.org/jira/browse/FLINK-9423 Project: Flink

[jira] [Closed] (FLINK-9064) Add Scaladocs link to documentation

2018-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9064. - Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Merged in: master: 

[jira] [Closed] (FLINK-9426) Harden RocksDBWriteBatchPerformanceTest.benchMark()

2018-05-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9426. - Resolution: Fixed Fix Version/s: 1.5.1 Merged in: master: b485f8cc60 release-1.5:

[jira] [Created] (FLINK-9436) Remove generic parameter namespace from InternalTimeServiceManager

2018-05-25 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9436: - Summary: Remove generic parameter namespace from InternalTimeServiceManager Key: FLINK-9436 URL: https://issues.apache.org/jira/browse/FLINK-9436 Project: Flink

[jira] [Closed] (FLINK-9571) Switch to internal states in StateBinder

2018-06-18 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9571. - Resolution: Implemented Merged in: master: 0bdde8377c > Switch to internal states in

[jira] [Closed] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

2018-06-13 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9506. - Resolution: Not A Problem > Flink ReducingState.add causing more than 100% performance drop >

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

2018-06-13 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510870#comment-16510870 ] Stefan Richter commented on FLINK-9506: --- [~yow] I would suggest that you discuss it on the user

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

2018-06-13 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510747#comment-16510747 ] Stefan Richter commented on FLINK-9506: --- Hi, I think this discussion has no connection with this

[jira] [Closed] (FLINK-9487) Prepare InternalTimerHeap for asynchronous snapshots

2018-06-15 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9487. - Resolution: Implemented Merged in: master: 7e0eafa74d > Prepare InternalTimerHeap for

[jira] [Closed] (FLINK-9601) Snapshot of CopyOnWriteStateTable will failed when the amount of record is more than MAXIMUM_CAPACITY

2018-06-18 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9601. - Resolution: Fixed Merged in: master: 0e9b066aab > Snapshot of CopyOnWriteStateTable will failed

[jira] [Assigned] (FLINK-9440) Allow cancelation and reset of timers

2018-05-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-9440: - Assignee: Stefan Richter > Allow cancelation and reset of timers >

[jira] [Closed] (FLINK-9423) Implement efficient deletes for heap based timer service

2018-05-31 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9423. - Resolution: Fixed Merged in: master: ff0b9c1eed > Implement efficient deletes for heap based

[jira] [Closed] (FLINK-9436) Remove generic parameter namespace from InternalTimeServiceManager

2018-05-31 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9436. - Resolution: Fixed Merged in: master: 57b950796d > Remove generic parameter namespace from

[jira] [Commented] (FLINK-9480) Let local recovery support rescaling

2018-05-30 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494897#comment-16494897 ] Stefan Richter commented on FLINK-9480: --- Maybe [~StephanEwen] or [~till.rohrmann] can also give

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-05-30 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494851#comment-16494851 ] Stefan Richter commented on FLINK-9268: --- Please notice that this is our attempt to "fix" (= for

[jira] [Commented] (FLINK-9480) Let local recovery support rescaling

2018-05-30 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494892#comment-16494892 ] Stefan Richter commented on FLINK-9480: --- Can you give some more details why you think this is

[jira] [Closed] (FLINK-9355) Simplify configuration of local recovery to a simple on/off

2018-05-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9355. - Resolution: Fixed Fix Version/s: 1.6.0 Merged in: master: 7f42259 release-1.5: b907af2ed6

[jira] [Commented] (FLINK-9440) Allow cancelation and reset of timers

2018-05-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492547#comment-16492547 ] Stefan Richter commented on FLINK-9440: --- Yes, was just about to answer to this. Once FLINK-9423 is

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

2018-06-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500165#comment-16500165 ] Stefan Richter commented on FLINK-9506: --- [~yow] I had another look at your code and can point out a

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

2018-06-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499947#comment-16499947 ] Stefan Richter commented on FLINK-9506: --- >From what I can see, the problem is purely related to

[jira] [Created] (FLINK-9485) Improving Flink’s timer management for large state

2018-06-01 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9485: - Summary: Improving Flink’s timer management for large state Key: FLINK-9485 URL: https://issues.apache.org/jira/browse/FLINK-9485 Project: Flink Issue

[jira] [Created] (FLINK-9487) Prepare InternalTimerHeap for asynchronous snapshots

2018-06-01 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9487: - Summary: Prepare InternalTimerHeap for asynchronous snapshots Key: FLINK-9487 URL: https://issues.apache.org/jira/browse/FLINK-9487 Project: Flink Issue

[jira] [Assigned] (FLINK-9486) Introduce TimerState in keyed state backend

2018-06-01 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-9486: - Assignee: Stefan Richter > Introduce TimerState in keyed state backend >

[jira] [Created] (FLINK-9486) Introduce TimerState in keyed state backend

2018-06-01 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9486: - Summary: Introduce TimerState in keyed state backend Key: FLINK-9486 URL: https://issues.apache.org/jira/browse/FLINK-9486 Project: Flink Issue Type:

[jira] [Commented] (FLINK-9486) Introduce TimerState in keyed state backend

2018-06-01 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497952#comment-16497952 ] Stefan Richter commented on FLINK-9486: --- Hi, I am already creating the issues for planning, but I

[jira] [Created] (FLINK-9489) Checkpoint timers as part of managed keyed state instead of raw keyed state

2018-06-01 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9489: - Summary: Checkpoint timers as part of managed keyed state instead of raw keyed state Key: FLINK-9489 URL: https://issues.apache.org/jira/browse/FLINK-9489 Project:

[jira] [Created] (FLINK-9491) Implement timer service based on RocksDB

2018-06-01 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9491: - Summary: Implement timer service based on RocksDB Key: FLINK-9491 URL: https://issues.apache.org/jira/browse/FLINK-9491 Project: Flink Issue Type:

[jira] [Created] (FLINK-9490) Provide backwards compatibility for timer state of Flink 1.5

2018-06-01 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9490: - Summary: Provide backwards compatibility for timer state of Flink 1.5 Key: FLINK-9490 URL: https://issues.apache.org/jira/browse/FLINK-9490 Project: Flink

[jira] [Updated] (FLINK-9490) Provide backwards compatibility for timer state of Flink 1.5

2018-06-01 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-9490: -- Fix Version/s: 1.6.0 Component/s: State Backends, Checkpointing > Provide backwards

[jira] [Updated] (FLINK-9491) Implement timer data structure based on RocksDB

2018-06-01 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-9491: -- Summary: Implement timer data structure based on RocksDB (was: Implement timer service based

[jira] [Closed] (FLINK-9440) Allow cancelation and reset of timers

2018-06-05 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9440. - Resolution: Fixed Fix Version/s: 1.6.0 Merged in: master: a0f4239fae > Allow cancelation

[jira] [Closed] (FLINK-8790) Improve performance for recovery from incremental checkpoint

2018-06-05 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8790. - Resolution: Fixed Merged in: master: bbf7ff2273 > Improve performance for recovery from

[jira] [Commented] (FLINK-9450) Job hangs if S3 access it denied during checkpoints

2018-05-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492587#comment-16492587 ] Stefan Richter commented on FLINK-9450: --- Do you have some logs for this problem, and/or a thread

[jira] [Comment Edited] (FLINK-9450) Job hangs if S3 access it denied during checkpoints

2018-05-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492587#comment-16492587 ] Stefan Richter edited comment on FLINK-9450 at 5/28/18 11:55 AM: - Do you

[jira] [Closed] (FLINK-9153) TaskManagerRunner should support rpc port range

2018-05-28 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9153. - Resolution: Fixed Fix Version/s: 1.6.0 Merged in: master: 7c90447849 release-1.5:

[jira] [Closed] (FLINK-7866) Weigh list of preferred locations for scheduling

2018-06-04 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-7866. - Resolution: Fixed Merged in: master: 8868ff5b05 > Weigh list of preferred locations for

[jira] [Created] (FLINK-9702) Improvement in (de)serialization of keys and values for RocksDB state

2018-07-02 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-9702: - Summary: Improvement in (de)serialization of keys and values for RocksDB state Key: FLINK-9702 URL: https://issues.apache.org/jira/browse/FLINK-9702 Project: Flink

[jira] [Commented] (FLINK-9702) Improvement in (de)serialization of keys and values for RocksDB state

2018-07-02 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529733#comment-16529733 ] Stefan Richter commented on FLINK-9702: --- I have a WIP branch that implements many of the

[jira] [Closed] (FLINK-7897) Consider using nio.Files for file deletion in TransientBlobCleanupTask

2018-06-26 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-7897. - Resolution: Implemented Merged in: master: 8674b69964 > Consider using nio.Files for file

[jira] [Closed] (FLINK-9263) Kafka010ITCase failed on travis because of the concurrency problem in DefaultOperateStateBackend

2018-05-02 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9263. - Resolution: Fixed Merged in: master: c11f11359b release 1.5: de4f283087 > Kafka010ITCase

[jira] [Commented] (FLINK-7484) CaseClassSerializer.duplicate() does not perform proper deep copy

2018-05-02 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460907#comment-16460907 ] Stefan Richter commented on FLINK-7484: --- [~joshlemer] this looks like a different problem to me,

[jira] [Closed] (FLINK-9270) Upgrade RocksDB to 5.11.3, and resolve concurrent test invocation problem of @RetryOnFailure

2018-05-02 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-9270. - Resolution: Won't Fix Won't fix until we figure out a way around the performance regression of

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-05-02 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461231#comment-16461231 ] Stefan Richter commented on FLINK-9268: --- I don't think that is the cause. The exception comes right

[jira] [Commented] (FLINK-9290) The job is unable to recover from a checkpoint

2018-05-02 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461167#comment-16461167 ] Stefan Richter commented on FLINK-9290: --- [~sihuazhou] yes, looks like it. > The job is unable to

[jira] [Commented] (FLINK-9291) Checkpoint failure (CIRCULAR REFERENCE:java.lang.NegativeArraySizeException)

2018-05-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462025#comment-16462025 ] Stefan Richter commented on FLINK-9291: --- Yes, this is basically a duplicate, just the access to the

[jira] [Issue Comment Deleted] (FLINK-9268) RockDB errors from WindowOperator

2018-05-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter updated FLINK-9268: -- Comment: was deleted (was: Maybe what your are looking for is using a window with aggregate

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-05-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462086#comment-16462086 ] Stefan Richter commented on FLINK-9268: --- Yes, you could have duplicates in AT_LEAST_ONCE. But if

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-05-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462068#comment-16462068 ] Stefan Richter commented on FLINK-9268: --- The 2GB limit actually applies on a per-key-per-window

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-05-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462072#comment-16462072 ] Stefan Richter commented on FLINK-9268: --- Maybe what your are looking for is using a window with

[jira] [Commented] (FLINK-9268) RockDB errors from WindowOperator

2018-05-03 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462164#comment-16462164 ] Stefan Richter commented on FLINK-9268: --- It is hard to make any assumptions about your job without

[jira] [Assigned] (FLINK-4809) Operators should tolerate checkpoint failures

2017-10-20 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-4809: - Assignee: Stefan Richter > Operators should tolerate checkpoint failures >

[jira] [Closed] (FLINK-5372) Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints()

2017-10-19 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-5372. - Resolution: Fixed Merged in dbf4c86 > Fix

[jira] [Commented] (FLINK-8413) Snapshot state of aggregated data is not maintained in flink's checkpointing

2018-01-11 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322191#comment-16322191 ] Stefan Richter commented on FLINK-8413: --- Can you share a (minimal) code example that shows this

[jira] [Comment Edited] (FLINK-8413) Snapshot state of aggregated data is not maintained in flink's checkpointing

2018-01-11 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322191#comment-16322191 ] Stefan Richter edited comment on FLINK-8413 at 1/11/18 1:25 PM: Can you

[jira] [Comment Edited] (FLINK-8411) inconsistent behavior between HeapListState#add() and RocksDBListState#add()

2018-01-11 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321968#comment-16321968 ] Stefan Richter edited comment on FLINK-8411 at 1/11/18 9:46 AM: Yes, I

[jira] [Commented] (FLINK-8411) inconsistent behavior between HeapListState#add() and RocksDBListState#add()

2018-01-11 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321968#comment-16321968 ] Stefan Richter commented on FLINK-8411: --- Yes, I agree. Would you like to make a PR or should I just

[jira] [Closed] (FLINK-7475) support update() in ListState

2018-01-10 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-7475. - Resolution: Fixed Merged in 438e4e3742. > support update() in ListState >

[jira] [Created] (FLINK-8385) Fix exceptions in AbstractEventTimeWindowCheckpointingITCase

2018-01-08 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-8385: - Summary: Fix exceptions in AbstractEventTimeWindowCheckpointingITCase Key: FLINK-8385 URL: https://issues.apache.org/jira/browse/FLINK-8385 Project: Flink

[jira] [Closed] (FLINK-7938) support addAll() in ListState

2018-01-19 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-7938. - Resolution: Implemented Merged in 14840809b1. > support addAll() in ListState >

[jira] [Commented] (FLINK-8487) State loss after multiple restart attempts

2018-01-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335551#comment-16335551 ] Stefan Richter commented on FLINK-8487: --- Afaik [~aljoscha] already fixed this in FLINK-7783? >

[jira] [Closed] (FLINK-8469) relocate and unify RocksDB option params in RocksDBPerformanceTest

2018-01-23 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-8469. - Resolution: Fixed Merged in 1c9c1e36c1dd7b3f2a160216e405302d7854c148 . > relocate and unify

<    1   2   3   4   5   6   7   8   9   10   >