[jira] [Created] (FLINK-11000) Introduce Resource Blacklist Mechanism

Yingjie Cao created FLINK-11000: --- Summary: Introduce Resource Blacklist Mechanism Key: FLINK-11000 URL: https://issues.apache.org/jira/browse/FLINK-11000 Project: Flink Issue Type: Improvement

[jira] [Assigned] (FLINK-11000) Introduce Resource Blacklist Mechanism

[ https://issues.apache.org/jira/browse/FLINK-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao reassigned FLINK-11000: --- Assignee: Yingjie Cao > Introduce Resource Blacklist Mechanism >

[jira] [Created] (FLINK-11859) Improve SpanningRecordSerializer performance by serializing record length to serialization buffer directly

Yingjie Cao created FLINK-11859: --- Summary: Improve SpanningRecordSerializer performance by serializing record length to serialization buffer directly Key: FLINK-11859 URL:

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819778#comment-16819778 ] Yingjie Cao commented on FLINK-12070: - I'd like to perform a test to compare the performance of old

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819813#comment-16819813 ] Yingjie Cao commented on FLINK-12070: - [~eaglewatcher] Indeed, disks are usually shared in a

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851502#comment-16851502 ] Yingjie Cao commented on FLINK-12070: - Because the test result is fairly long, I posted it in google

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849615#comment-16849615 ] Yingjie Cao commented on FLINK-12070: - I am now testing the latest implementation, and find some

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855260#comment-16855260 ] Yingjie Cao commented on FLINK-12070: - [~StephanEwen] There is no swap partition in my testing OS.

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861863#comment-16861863 ] Yingjie Cao commented on FLINK-12070: - [~StephanEwen] The total memory of network buffer pool per

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860643#comment-16860643 ] Yingjie Cao commented on FLINK-12070: - [~StephanEwen] Last week, I have implemented several new ways

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872295#comment-16872295 ] Yingjie Cao commented on FLINK-12070: - [~StephanEwen] That will be cool if you can turn the branch

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872281#comment-16872281 ] Yingjie Cao commented on FLINK-12070: - [~Zentol] OK, I'd like to open a PR. Based on the previous 

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869404#comment-16869404 ] Yingjie Cao commented on FLINK-12070: - I have appended test results of the new implementation

[jira] [Commented] (FLINK-13100) Fix the unexpected IOException during FileBufferReader#nextBuffer

[ https://issues.apache.org/jira/browse/FLINK-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881129#comment-16881129 ] Yingjie Cao commented on FLINK-13100: - The deadlock problem of spillable subpartition has been

[jira] [Created] (FLINK-12329) Netty thread deadlock bug of the SpilledSubpartitionView

Yingjie Cao created FLINK-12329: --- Summary: Netty thread deadlock bug of the SpilledSubpartitionView Key: FLINK-12329 URL: https://issues.apache.org/jira/browse/FLINK-12329 Project: Flink Issue

[jira] [Comment Edited] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825700#comment-16825700 ] Yingjie Cao edited comment on FLINK-12070 at 4/25/19 4:18 AM: -- The test is

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825700#comment-16825700 ] Yingjie Cao commented on FLINK-12070: - The test is still running and will take a bit more time. Now

[jira] [Comment Edited] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825700#comment-16825700 ] Yingjie Cao edited comment on FLINK-12070 at 4/25/19 4:21 AM: -- The test is

[jira] [Comment Edited] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825700#comment-16825700 ] Yingjie Cao edited comment on FLINK-12070 at 4/25/19 4:25 AM: -- The test is

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872870#comment-16872870 ] Yingjie Cao commented on FLINK-12070: - [~StephanEwen] Though, theoretically, two read buffers is

[jira] [Updated] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-13489: Labels: pull-request-available (was: ) > Heavy deployment end-to-end test fails on Travis with

[jira] [Comment Edited] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899343#comment-16899343 ] Yingjie Cao edited comment on FLINK-13489 at 8/3/19 3:34 AM: - [~StephanEwen]

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899343#comment-16899343 ] Yingjie Cao commented on FLINK-13489: - [~StephanEwen] I run the test for many times, but only

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898882#comment-16898882 ] Yingjie Cao commented on FLINK-13489: - I'd like to open a PR and fix it. > Heavy deployment

[jira] [Created] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush

Yingjie Cao created FLINK-14118: --- Summary: Reduce the unnecessary flushing when there is no data available for flush Key: FLINK-14118 URL: https://issues.apache.org/jira/browse/FLINK-14118 Project:

[jira] [Updated] (FLINK-11859) Improve SpanningRecordSerializer performance by serializing record length to serialization buffer directly

[ https://issues.apache.org/jira/browse/FLINK-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-11859: Fix Version/s: 1.10.0 > Improve SpanningRecordSerializer performance by serializing record length

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899860#comment-16899860 ] Yingjie Cao commented on FLINK-13489: - [~carp84] Thanks for reminding, I have marked the status as

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899957#comment-16899957 ] Yingjie Cao commented on FLINK-13489: - The GC time is still high after I Increased the heap size of

[jira] [Comment Edited] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899957#comment-16899957 ] Yingjie Cao edited comment on FLINK-13489 at 8/5/19 9:50 AM: - The GC time is

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901841#comment-16901841 ] Yingjie Cao commented on FLINK-13489: - [~till.rohrmann] Is there a Jira for the tpch end-to-end test 

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901830#comment-16901830 ] Yingjie Cao commented on FLINK-13489: - [~till.rohrmann] I am still working on this issue, could you

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901970#comment-16901970 ] Yingjie Cao commented on FLINK-13489: - [~till.rohrmann] I agree with you, this should not be a

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902750#comment-16902750 ] Yingjie Cao commented on FLINK-13489: - I ran the heavy deployment test for many times and found the

[jira] [Updated] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-13489: Attachment: image-2019-08-01-18-53-21-966.png > Heavy deployment end-to-end test fails on Travis

[jira] [Updated] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-13489: Attachment: (was: image-2019-08-01-18-53-21-966.png) > Heavy deployment end-to-end test fails

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898497#comment-16898497 ] Yingjie Cao commented on FLINK-13489: - I tested different setting on Travis for the heavy deployment

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903757#comment-16903757 ] Yingjie Cao commented on FLINK-13489: - [~till.rohrmann] I'll try to reproduce the problem and figure

[jira] [Commented] (FLINK-13489) Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout

[ https://issues.apache.org/jira/browse/FLINK-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897985#comment-16897985 ] Yingjie Cao commented on FLINK-13489: - [~tzulitai] I wonder how this test was triggered? I used both

[jira] [Commented] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush

[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933359#comment-16933359 ] Yingjie Cao commented on FLINK-14118: - I couldn't come up with any bad case theoretically.  A new

[jira] [Commented] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

[ https://issues.apache.org/jira/browse/FLINK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938472#comment-16938472 ] Yingjie Cao commented on FLINK-14195: - Thanks [~trohrmann]. My branch is 

[jira] [Commented] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

[ https://issues.apache.org/jira/browse/FLINK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938383#comment-16938383 ] Yingjie Cao commented on FLINK-14195: - Hi [~trohrmann], I am trying to fix the problem but

[jira] [Commented] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938406#comment-16938406 ] Yingjie Cao commented on FLINK-12576: - [~alpinegizmo] Because only one of the two upstream task will

[jira] [Closed] (FLINK-14195) Nightly connectors-jdk11 fails because of missing jaxb classes

[ https://issues.apache.org/jira/browse/FLINK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-14195. --- > Nightly connectors-jdk11 fails because of missing jaxb classes >

[jira] [Commented] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush

[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935947#comment-16935947 ] Yingjie Cao commented on FLINK-14118: - [~pnowojski] You are right. High data skew can reproduce the

[jira] [Comment Edited] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush

[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935947#comment-16935947 ] Yingjie Cao edited comment on FLINK-14118 at 9/23/19 3:17 PM: -- [~pnowojski] 

[jira] [Comment Edited] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush

[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935947#comment-16935947 ] Yingjie Cao edited comment on FLINK-14118 at 9/23/19 3:13 PM: -- [~pnowojski] 

[jira] [Created] (FLINK-14185) AbstractServerTest failed on Travis

Yingjie Cao created FLINK-14185: --- Summary: AbstractServerTest failed on Travis Key: FLINK-14185 URL: https://issues.apache.org/jira/browse/FLINK-14185 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-14139) Fix potential memory leak of rest server when using session/standalone cluster

Yingjie Cao created FLINK-14139: --- Summary: Fix potential memory leak of rest server when using session/standalone cluster Key: FLINK-14139 URL: https://issues.apache.org/jira/browse/FLINK-14139

[jira] [Commented] (FLINK-14139) Fix potential memory leak of rest server when using session/standalone cluster

[ https://issues.apache.org/jira/browse/FLINK-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934144#comment-16934144 ] Yingjie Cao commented on FLINK-14139: - Some other user of netty also encountered this problem: 

[jira] [Commented] (FLINK-14118) Reduce the unnecessary flushing when there is no data available for flush

[ https://issues.apache.org/jira/browse/FLINK-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934201#comment-16934201 ] Yingjie Cao commented on FLINK-14118: - I'd like to monitor the behavior of upgraded jobs and report

[jira] [Updated] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-12576: Attachment: image-2019-09-26-11-34-24-878.png > inputQueueLength metric does not work for

[jira] [Updated] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-12576: Attachment: image-2019-09-26-11-36-06-027.png > inputQueueLength metric does not work for

[jira] [Commented] (FLINK-12576) inputQueueLength metric does not work for LocalInputChannels

[ https://issues.apache.org/jira/browse/FLINK-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938232#comment-16938232 ] Yingjie Cao commented on FLINK-12576: - [~alpinegizmo] I tried the instruction gave above, but the

[jira] [Created] (FLINK-14180) Enable config of maximum capacity of FileArchivedExecutionGraphStore.

Yingjie Cao created FLINK-14180: --- Summary: Enable config of maximum capacity of FileArchivedExecutionGraphStore. Key: FLINK-14180 URL: https://issues.apache.org/jira/browse/FLINK-14180 Project: Flink

[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times

[ https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863930#comment-16863930 ] Yingjie Cao commented on FLINK-12070: - For the benchmark, I use the same taskmanager setting for all

[jira] [Commented] (FLINK-13100) Fix the bug of throwing IOException while FileBufferReader#nextBuffer

[ https://issues.apache.org/jira/browse/FLINK-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881859#comment-16881859 ] Yingjie Cao commented on FLINK-13100: - Glad to see the bug will be fix in version 1.9, thanks

[jira] [Closed] (FLINK-14498) Introduce NetworkBufferPool#isAvailable() for non-blocking output

[ https://issues.apache.org/jira/browse/FLINK-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-14498. --- > Introduce NetworkBufferPool#isAvailable() for non-blocking output >

[jira] [Commented] (FLINK-14603) NetworkBufferPoolTest.testBlockingRequestFromMultiLocalBufferPool timeout in travis

[ https://issues.apache.org/jira/browse/FLINK-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967377#comment-16967377 ] Yingjie Cao commented on FLINK-14603: - Thanks, [~trohrmann] and [~ykt836]. I will try to find out

[jira] [Closed] (FLINK-14603) NetworkBufferPoolTest.testBlockingRequestFromMultiLocalBufferPool timeout in travis

[ https://issues.apache.org/jira/browse/FLINK-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-14603. --- > NetworkBufferPoolTest.testBlockingRequestFromMultiLocalBufferPool timeout in > travis >

[jira] [Updated] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14872: Fix Version/s: 1.10.0 > Potential deadlock for task reading from blocking ResultPartition. >

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979037#comment-16979037 ] Yingjie Cao commented on FLINK-14872: - Only limit the number of Buffers can be allocated can only

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979002#comment-16979002 ] Yingjie Cao commented on FLINK-14845: - [~pnowojski] , thanks for proposing these options. #

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979069#comment-16979069 ] Yingjie Cao commented on FLINK-14872: - Theoretically, the problem is also exist with Pipelined mode.

[jira] [Commented] (FLINK-14818) StreamNetworkBenchmarkEnvironment incorrectly setups a receiving InputGate

[ https://issues.apache.org/jira/browse/FLINK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979219#comment-16979219 ] Yingjie Cao commented on FLINK-14818: - I found there seems no very straightforward way to achieve

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979191#comment-16979191 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] They are relevant and the causes are similar

[jira] [Updated] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14845: Fix Version/s: 1.10.0 > Introduce data compression to blocking shuffle. >

[jira] [Created] (FLINK-15024) System classloader memory leak after loading too many codegen classes.

Yingjie Cao created FLINK-15024: --- Summary: System classloader memory leak after loading too many codegen classes. Key: FLINK-15024 URL: https://issues.apache.org/jira/browse/FLINK-15024 Project: Flink

[jira] [Created] (FLINK-15030) Potential deadlock for bounded blocking ResultPartition.

Yingjie Cao created FLINK-15030: --- Summary: Potential deadlock for bounded blocking ResultPartition. Key: FLINK-15030 URL: https://issues.apache.org/jira/browse/FLINK-15030 Project: Flink Issue

[jira] [Updated] (FLINK-15024) System classloader memory leak after loading too many codegen classes.

[ https://issues.apache.org/jira/browse/FLINK-15024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-15024: Description: We are using Flink session cluster as a service for ad-hoc queries. After running

[jira] [Created] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

Yingjie Cao created FLINK-15140: --- Summary: Shuffle data compression does not work with BroadcastRecordWriter. Key: FLINK-15140 URL: https://issues.apache.org/jira/browse/FLINK-15140 Project: Flink

[jira] [Updated] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-15140: Description: I tested the newest code of master branch last weekend with more test cases.

[jira] [Closed] (FLINK-15166) Shuffle data compression wrongly decrease the buffer reference count.

[ https://issues.apache.org/jira/browse/FLINK-15166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-15166. --- > Shuffle data compression wrongly decrease the buffer reference count. >

[jira] [Resolved] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-15140. - Resolution: Fixed > Shuffle data compression does not work with BroadcastRecordWriter. >

[jira] [Commented] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995511#comment-16995511 ] Yingjie Cao commented on FLINK-15140: - Fix via 5139282e7f48ece00528c6ade9480f10fa3dd54b on master.

[jira] [Closed] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-15140. --- > Shuffle data compression does not work with BroadcastRecordWriter. >

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988808#comment-16988808 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] You are right. We may add a related link to this

[jira] [Commented] (FLINK-14952) Yarn containers can exceed physical memory limits when using BoundedBlockingSubpartition.

[ https://issues.apache.org/jira/browse/FLINK-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989448#comment-16989448 ] Yingjie Cao commented on FLINK-14952: - [~gjy] By "FILE-FILE" mode, I mean using

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989723#comment-16989723 ] Yingjie Cao commented on FLINK-14872: - Sadly, I find the number of exclusive buffer a channel

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989755#comment-16989755 ] Yingjie Cao commented on FLINK-14872: - By the way, I will test the change by running tpc-ds

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989800#comment-16989800 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] I mean creating a zero size buffer pool for

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989957#comment-16989957 ] Yingjie Cao commented on FLINK-14872: - As discussed, we would implement a temporary fix for the

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990532#comment-16990532 ] Yingjie Cao commented on FLINK-14872: - [~pnowojski] Thanks for the review. I have already tested the

[jira] [Resolved] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-14845. - Release Note: Now user can set taskmanager.network.blocking-shuffle.compression.enabled to true

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990697#comment-16990697 ] Yingjie Cao commented on FLINK-14845: - Fix via 66d4d7da2d8b717f420509d9785fad0880562f10 on master.

[jira] [Commented] (FLINK-15239) TM Metaspace memory leak

[ https://issues.apache.org/jira/browse/FLINK-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997100#comment-16997100 ] Yingjie Cao commented on FLINK-15239: - Not only TM, JM also suffers from the problem. > TM

[jira] [Commented] (FLINK-14498) Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool

[ https://issues.apache.org/jira/browse/FLINK-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960764#comment-16960764 ] Yingjie Cao commented on FLINK-14498: - [~wind_ljy] The PR is based on another PR 

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978153#comment-16978153 ] Yingjie Cao commented on FLINK-14845: - To verify how many improvements we could gain from

[jira] [Created] (FLINK-14845) Introduce data compression to blocking shuffle.

Yingjie Cao created FLINK-14845: --- Summary: Introduce data compression to blocking shuffle. Key: FLINK-14845 URL: https://issues.apache.org/jira/browse/FLINK-14845 Project: Flink Issue Type:

[jira] [Updated] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14845: Description: Currently, blocking shuffle writer writes raw output data to disk without

[jira] [Updated] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14872: Description: Currently, the buffer pool size of InputGate reading from blocking ResultPartition

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978184#comment-16978184 ] Yingjie Cao commented on FLINK-14845: - [~lzljs3620320] For small data, for example, dozens of byte,

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978288#comment-16978288 ] Yingjie Cao commented on FLINK-14845: - [~chesnay] A choice maybe make the compression codec cluster

[jira] [Created] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

Yingjie Cao created FLINK-14872: --- Summary: Potential deadlock for task reading from blocking ResultPartition. Key: FLINK-14872 URL: https://issues.apache.org/jira/browse/FLINK-14872 Project: Flink

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978284#comment-16978284 ] Yingjie Cao commented on FLINK-14845: - [~pnowojski]  You are right, taking both Blocking and

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978241#comment-16978241 ] Yingjie Cao commented on FLINK-14845: - Doing compression and decompression with task threads may be

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982271#comment-16982271 ] Yingjie Cao commented on FLINK-14845: - > Where are you going to compress data into? Each compressor

[jira] [Commented] (FLINK-14952) Yarn containers can exceed physical memory limits when using BoundedBlockingSubpartition.

[ https://issues.apache.org/jira/browse/FLINK-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983397#comment-16983397 ] Yingjie Cao commented on FLINK-14952: - Can we manage the memory mapped region and restrict its size?

[jira] [Updated] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-14845: Labels: pull-request-available (was: ) > Introduce data compression to blocking shuffle. >

[jira] [Commented] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

[ https://issues.apache.org/jira/browse/FLINK-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985948#comment-16985948 ] Yingjie Cao commented on FLINK-14872: - > If we had to implement something, I would be voting more

[jira] [Commented] (FLINK-14845) Introduce data compression to blocking shuffle.

[ https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985873#comment-16985873 ] Yingjie Cao commented on FLINK-14845: - Based on the above discussions, I have given an

  1   2   3   4   5   6   7   8   >