Hi, Thanks, that's the way I did it, it was just that I thought for a second about a more complicated system property of the -D kind, but the normal way is enough for my current case, thanks.
Ismaël On Fri, May 20, 2016 at 2:56 PM, Aljoscha Krettek <[email protected]> wrote: > Hi, > you should be able to set the runner using command line options, e.g: > --runner=org.apache.beam.runners.flink.FlinkPipelineRunner > --streaming=false --parallelism=8 > > And in the program have something like: > Options options = PipelineOptionsFactory.fromArgs(args).withValidation(); > > -Aljoscha > > > On Fri, 20 May 2016 at 13:59 Jean-Baptiste Onofré <[email protected]> wrote: > >> You can always do it via System, populating the options, right ? >> >> Regards >> JB >> >> On 05/20/2016 11:07 AM, Ismaël Mejía wrote: >> > Hello again, >> > >> > I just executed our pipeline with the changes you just merged, and guess >> > what ? It works perfectly (in batch mode), amazing job guys. I have >> just one >> > extra question, is there a way to configure the pipeline options with >> system >> > properties (-D...). e.g. to configure the streaming/batch mode and >> > avoid to do >> > this explicitly in the code (to remove the explicit FlinkPipelineOptions >> > import) ? >> > >> > Ismaël >> > >> > >> > On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi Ismael, >> > >> > Side inputs are not supported when the Flink Runner runs in >> streaming >> > mode. Actually, we just merged changes which allow you to use the >> > batch mode (streaming set to false) with Windows and side inputs. >> > >> > So just checkout the latest Beam master and rebuild. Set the >> streaming >> > flag to false again and rerun your example. >> > >> > Thanks, >> > Max >> > >> > On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <[email protected] >> > <mailto:[email protected]>> wrote: >> > > >> > > Hello, >> > > >> > > Thanks for your answer, I am really happy to know you are >> working on >> > > removing >> > > this flag for batch/streaming since this is not really Beam-like. >> > > >> > > I tried to execute the pipeline again in streaming mode, and I >> > got this >> > > exception now: >> > > >> > > java.lang.UnsupportedOperationException: The transform >> > > View.CreatePCollectionView is currently not supported. >> > > >> > > Any info about this one ? >> > > >> > > Thanks again, >> > > Ismael >> > > >> > > >> > > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek >> > <[email protected] <mailto:[email protected]>> >> > > wrote: >> > >> >> > >> Hi, >> > >> as an addition, there are plans to get rid of the "streaming" >> > flag for all >> > >> runners and determine the mode from the job. >> > >> >> > >> -Aljoscha >> > >> >> > >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <[email protected] >> > <mailto:[email protected]>> wrote: >> > >>> >> > >>> Hi Ismaël, >> > >>> >> > >>> There is a pull request which enables Windows on batch. It >> will be >> > >>> merged soon. In the meantime, please use the streaming >> > execution mode >> > >>> of the Flink Runner. >> > >>> >> > >>> Use setStreaming(true) to enable it on the >> FlinkPipelineOptions. >> > >>> >> > >>> Cheers, >> > >>> Max >> > >>> >> > >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía >> > <[email protected] <mailto:[email protected]>> wrote: >> > >>> > I was trying to run a local Pipeline we have (basically a >> > batch one we >> > >>> > rewrote from TPC-H) in all the possible runners, for the >> > moment I had >> > >>> > success with the DirectPipelineRunner, >> > InProcessPipelineRunner and >> > >>> > SparkPipelineRunner. However when I executed it in the >> > >>> > FlinkPipelineRunner I >> > >>> > got this exception: >> > >>> > >> > >>> > java.lang.UnsupportedOperationException: The transform >> > Window.Into() is >> > >>> > currently not supported. >> > >>> > >> > >>> > I thought it was probably a configuration error since I >> > remember that >> > >>> > in the >> > >>> > flink runner we had to explicitly define if it was a batch or >> > streaming >> > >>> > job, >> > >>> > but when I checked the options: >> > >>> > >> > >>> > Current Settings: >> > >>> > appName: TestQ08Task >> > >>> > filesToStage: ... >> > >>> > flinkMaster: [auto] >> > >>> > parallelism: 1 >> > >>> > runner: class >> org.apache.beam.runners.flink.FlinkPipelineRunner >> > >>> > stableUniqueNames: WARNING >> > >>> > streaming: false >> > >>> > >> > >>> > I noticed I was in batch mode, is there something missing in >> the >> > >>> > options >> > >>> > configuration, or is it that the UnsoppertedOperation is a >> > WIP or not >> > >>> > supported at all by Flink ? >> > >>> > >> > >>> > And extra question, this option to define if it is in batch >> > or stream >> > >>> > mode >> > >>> > is going to stay for long, can't be this inferred some how ? >> > >>> > >> > >>> > Thanks, >> > >>> > -Ismaël >> > >>> > >> > > >> > > >> > >> > >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
