[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-18 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766646#comment-17766646 ] Yangze Guo commented on FLINK-33053: master: d5e151f72336abcc13082fe4bb3e05fd5a785e86 > Watcher

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764950#comment-17764950 ] Zili Chen commented on FLINK-33053: --- But it's possible to add an option to explicitly identify the

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764948#comment-17764948 ] Zili Chen commented on FLINK-33053: --- No. Both {{CuratorCache}} and {{TreeCache}} doesn't "own" the

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764943#comment-17764943 ] Yangze Guo commented on FLINK-33053: Thanks for the pointer [~tison] . I'd like to add a safetynet

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764723#comment-17764723 ] Zili Chen commented on FLINK-33053: --- But we don't have other shared watchers so we can force remove

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764721#comment-17764721 ] Zili Chen commented on FLINK-33053: --- See

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764702#comment-17764702 ] Zili Chen commented on FLINK-33053: --- I noticed that the {{TreeCache}}'s close call {{removeWatches}}

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-13 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764640#comment-17764640 ] Yangze Guo commented on FLINK-33053: JFYI, I'm still investigating the root cause of this, but I

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-11 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763713#comment-17763713 ] Yangze Guo commented on FLINK-33053: I just find that this issue is very easy to reproduce. To

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-09 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763315#comment-17763315 ] Yangze Guo commented on FLINK-33053: [~tison] Yes, it seems teh curator framework issued the watcher

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762965#comment-17762965 ] Zili Chen commented on FLINK-33053: --- The log seems trimed. I saw: 2023-09-08 11:09:03,738 DEBUG

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762962#comment-17762962 ] Matthias Pohl commented on FLINK-33053: --- Thanks for sharing this. I guess, the thread dump (which

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762936#comment-17762936 ] Yangze Guo commented on FLINK-33053: I reproduce the issue and filter out some logs. [^26.dump.zip]

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762700#comment-17762700 ] Yangze Guo commented on FLINK-33053: Thanks for the pointer [~mapohl]. I'll try to get a debug log

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762680#comment-17762680 ] Matthias Pohl commented on FLINK-33053: --- FLINK-29813 is already covering the migration to

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762677#comment-17762677 ] Yangze Guo commented on FLINK-33053: [~tison] Could we use CuratorCache instead? Would it be more

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762657#comment-17762657 ] Zili Chen commented on FLINK-33053: --- Perhaps you can enable debug logs and check "Removing watcher for

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Zili Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762599#comment-17762599 ] Zili Chen commented on FLINK-33053: --- The recipe in use is {{TreeCache}}, which doesn't change from

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762582#comment-17762582 ] Matthias Pohl commented on FLINK-33053: --- Can you share the logs and the thread dump of this run?

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762579#comment-17762579 ] Yangze Guo commented on FLINK-33053: In our test, the log shows the ZooKeeperLeaderRetrievalDriver

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-07 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762578#comment-17762578 ] Yangze Guo commented on FLINK-33053: [~mapohl] Thanks for your help. I check it with curator 5.5.0

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-06 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762574#comment-17762574 ] Matthias Pohl commented on FLINK-33053: --- Thanks for bringing this up, [~guoyangze]. I'm going to

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-06 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762560#comment-17762560 ] Yangze Guo commented on FLINK-33053: Also cc [~wangyang0918] > Watcher leak in Zookeeper HA mode >

[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode

2023-09-06 Thread Yangze Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762558#comment-17762558 ] Yangze Guo commented on FLINK-33053: [~mapohl] Would you like to take a look? > Watcher leak in