Hi, As JB mentioned - the current behavior of Spark runner is: Pipelines which use only bounded sources will exit immediately. Pipelines which contain 1 or more unbounded sources will not exit unless you specify some timeout.
A few questions: 1. Which version of Beam are you using? Have you tried with 2.0.0? WordCount example using 2.0.0 exits normally (since this pipeline only uses a bounded source - TextIO). 2. Are there any other sources in your pipeline other than TextIO? If there are any unbounded sources in the pipeline it will not exit unless you specify a timeout. 3. What pipeline options did you set? On Thu, May 11, 2017 at 10:05 AM bluejoe <[email protected]> wrote: > I used a bounded source: > TextIO.Read.from("./demo.txt”); > > Timeout is not configured > > I am confused why the waitUntilFinish() is not finished while the spark > job is actually done. > > Best regards, > bluejoe > > > > 在 2017/5/10 下午7:52，“Jean-Baptiste Onofré”<user-return-1537-bluejoe= > [email protected] 代表 [email protected]> 写入: > > >Hi, > > > >What's your source (unbounded or bounded) ? > >Did you define a timeout ? > > > >Regards > >JB > > > >On 05/10/2017 01:18 PM, bluejoe wrote: > >> Hi, > >> I create a beam pipeline in a Scala program, using the SparkRunner as > runner > >> with sparkMaster=local[4] > >> > >> However, when executing the following code: > >> > >> pipeline.run().waitUntilFinish(); > >> > >> > >> The program does not terminate even if the console displays: > >> > >> > >> 17/05/10 17:45:40 INFO SparkContext: Successfully stopped SparkContext > >> > >> 17/05/10 17:45:40 INFO RemoteActorRefProvider$RemotingTerminator: > Shutting down > >> remote daemon. > >> > >> 17/05/10 17:45:40 INFO RemoteActorRefProvider$RemotingTerminator: > Remote daemon > >> shut down; proceeding with flushing remote transports. > >> > >> 17/05/10 17:45:40 INFO RemoteActorRefProvider$RemotingTerminator: > Remoting shut > >> down. > >> > >> > >> > >> If I use pipeline in a right way? > >> Can anybody tell me how to let the program terminate normally? > >> > >> Best regards, > >> bluejoe > > > >-- > >Jean-Baptiste Onofré > >[email protected] > >http://blog.nanthrax.net > >Talend - http://www.talend.com > > >
