[ https://issues.apache.org/jira/browse/BEAM-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807065#comment-16807065 ]
Jiayi Zhao edited comment on BEAM-6956 at 4/1/19 9:41 PM: ---------------------------------------------------------- it might due to some random stuff, I tried in thread flink again with 100 worker_threads, and this time it works(I also changed -parallelism=1 instead of -parallelism=2 before, not sure if it's related), but when I tried separate local flink cluster, it works for some simple pipeline but got stuck for some complex tensorflow transform pipelines was (Author: 1025kb): it might due to some random stuff, I tried in thread flink again with 100 worker_threads, and this time it works(I also changed --parallelism=1 instead of --parallelism=2 before, not sure if it's related), when I tried separate local flink cluster, each time pipeline stuck at different places, I will try confirm it and put more information about where it stucks > --experiments=worker_threads=100 issue > -------------------------------------- > > Key: BEAM-6956 > URL: https://issues.apache.org/jira/browse/BEAM-6956 > Project: Beam > Issue Type: New Feature > Components: runner-flink > Reporter: Jiayi Zhao > Priority: Major > > I noticed that without this "–experiments=worker_threads=100", pipeline will > stuck, > The weird thing is, I tried some complex pipeline using the in thread flink > method (./gradlew :beam-runners-flink_2.11-job-server:runShadow) > "–experiments=worker_threads=100" doesn't work, but > "–experiments=worker_threads=1000" works fine > Then I tried the same pipeline using the separate local flink cluster > (./gradlew :beam-runners-flink_2.11-job-server:runShadow > -PflinkMasterUrl=localhost:8081), flink version is 1.5.6 (other version > doesn't work, see BEAM-6915) > Neither did "–experiments=worker_threads=1000" or > "–experiments=worker_threads=10000" work, pipeline stuck at certain stage > (shows running in flink UI but won't finish forever) > any real fix to that? Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)