[ 
https://issues.apache.org/jira/browse/BEAM-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807065#comment-16807065
 ] 

Jiayi Zhao edited comment on BEAM-6956 at 4/1/19 9:41 PM:
----------------------------------------------------------

it might due to some random stuff, I tried in thread flink again with 100 
worker_threads, and this time it works(I also changed -parallelism=1 instead of 
-parallelism=2 before, not sure if it's related),

but when I tried separate local flink cluster, it works for some simple 
pipeline but got stuck for some complex tensorflow transform pipelines

 


was (Author: 1025kb):
it might due to some random stuff, I tried in thread flink again with 100 
worker_threads, and this time it works(I also changed --parallelism=1 instead 
of --parallelism=2 before, not sure if it's related),

when I tried separate local flink cluster, each time pipeline stuck at 
different places, I will try confirm it and put more information about where it 
stucks 

> --experiments=worker_threads=100 issue
> --------------------------------------
>
>                 Key: BEAM-6956
>                 URL: https://issues.apache.org/jira/browse/BEAM-6956
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-flink
>            Reporter: Jiayi Zhao
>            Priority: Major
>
> I noticed that without this "–experiments=worker_threads=100", pipeline will 
> stuck,
> The weird thing is, I tried some complex pipeline using the in thread flink 
> method (./gradlew :beam-runners-flink_2.11-job-server:runShadow)
> "–experiments=worker_threads=100" doesn't work, but 
> "–experiments=worker_threads=1000" works fine
> Then I tried the same pipeline using the separate local flink cluster 
> (./gradlew :beam-runners-flink_2.11-job-server:runShadow 
> -PflinkMasterUrl=localhost:8081), flink version is 1.5.6 (other version 
> doesn't work, see BEAM-6915)
> Neither did "–experiments=worker_threads=1000" or 
> "–experiments=worker_threads=10000" work, pipeline stuck at certain stage 
> (shows running in flink UI but won't finish forever)
> any real fix to that? Thanks!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to