Re: Lateness for Spark

2018-09-12 Thread Amit Sela
event time triggers (via watermarks) should be supported as well. On Sun, Sep 9, 2018 at 11:57 PM Vishwas Bm wrote: > Hi, > > Thanks for the reply. As per the beam capability matrix only > Processing-time triggers is supported by spark runner. > As this page is not updated, what other triggers

Re: Lateness for Spark

2018-09-09 Thread Amit Sela
I don't think the capability matrix is updated, the Spark runner uses LateDataUtils to handle late elements - https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java#L300 On Fri, Sep 7, 2018 at 6:43 PM

Re: automatic runner inference

2017-04-05 Thread Amit Sela
I agree that the Spark runner should support submitting programatically to cluster, though not instead of "spark-submit" but in addition, so that Beam users enjoy a good experience of portability while Spark users enjoy a quick ramp-up into Beam. You can follow:

Re: Slack

2017-03-13 Thread Amit Sela
I'm so well trained, I do it on my phone now! On Mon, Mar 13, 2017, 15:24 Tobias Feldhaus <tobias.feldh...@localsearch.ch> wrote: > Same for me please :) > > Tobi > > > > On 13.03.17, 13:30, "Amit Sela" <amitsel...@gmail.com> wrote: > > > >

Re: Slack

2017-03-13 Thread Amit Sela
Done. Welcome! On Mon, Mar 13, 2017 at 2:29 PM Alexander Gallego <gallego.al...@gmail.com> wrote: > same for me please. > > > > > .alex > > > > On Fri, Mar 10, 2017 at 3:01 PM, Amit Sela <amitsel...@gmail.com> wrote: > > Done > >

Re: Regarding Beam Slack channel

2017-03-10 Thread Amit Sela
Done. Welcome! On Fri, Mar 10, 2017, 12:30 Borisa Zivkovic wrote: > Hi, > > can someone please add me to beam slack channel? > > thanks > Borisa >

Re: Monitoring and Management Tools for Beam an Friends

2017-03-02 Thread Amit Sela
+Stas Levin On Thu, Mar 2, 2017 at 5:30 PM Jean-Baptiste Onofré wrote: Hi Benjamin, It's a bit related to the Metric discussion on the dev@ mailing list. Today, we leverage the monitoring and management provided by the execution engine of the runner.

Re: collect to local

2017-02-20 Thread Amit Sela
Spark runner's EvaluationContext has a hook ready for this - but clearly only for batch, in streaming this feature doesn't seem relevant. You can easily

Re: Regarding Beam Slack Channel

2017-01-23 Thread Amit Sela
Done. Welcome! On Mon, Jan 23, 2017 at 11:29 PM Kai Jiang wrote: > Hi Amit, > > I am a new contributor. Could you add me as well? jiang...@gmail.com > > Thanks, > Kai > > On Mon, Jan 23, 2017 at 1:04 PM, Davor Bonaci wrote: > > I wouldn't be in favor of

Re: Regarding Beam Slack Channel

2017-01-23 Thread Amit Sela
Done. Welcome! On Mon, Jan 23, 2017 at 10:26 PM Wyatt Frelot <wjfr...@gmail.com> wrote: > Can I be added as well? > > wjfr...@gmail.com > > Wyatt > > On Jan 23, 2017 3:20 PM, "Amit Sela" <amitsel...@gmail.com> wrote: > > Done. Welcome! &g

Re: Regarding Beam Slack Channel

2017-01-23 Thread Amit Sela
Done. Welcome! On Mon, Jan 23, 2017 at 10:10 PM Chaoran Yu wrote: > Can anyone add me to the channel as well? Thank you! > chaoran...@lightbend.com > > Chaoran > > On Jan 23, 2017, at 1:32 AM, Ritesh Kasat wrote: > > Hello, > > Can someone add

Re: Beam Spark/Flink runner with DC/OS

2017-01-22 Thread Amit Sela
I'lll join JB's comment on the Spark runner saying that submitting Beam pipelines using the Spark runner can be done using Spark's spark-submit script, find out more in the Spark runner documentation . Amit. On Sun, Jan 22, 2017 at 8:03 AM

Re: Streaming job with Kafka on SparkRunner

2017-01-21 Thread Amit Sela
once before the first microbatch, and keep alive. Please correct me if any misunderstanding. Mingmin On Sat, Jan 21, 2017 at 1:35 AM, Amit Sela <amitsel...@gmail.com> wrote: Not sure why this would cause the application to crash, but I can give some background about how the Spark runner reads

Re: Streaming job with Kafka on SparkRunner

2017-01-21 Thread Amit Sela
ovider = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > ssl.keystore.location = null > ssl.cipher.suites = null > security.protocol = PLAINTEXT > ssl.keymanager.algorithm = SunX509 > metrics.sample.window.ms = 3 > sasl.callback.handler.class = null >

Re: Streaming job with Kafka on SparkRunner

2017-01-20 Thread Amit Sela
The WakeupException is being logged and not thrown (it is OK since the reader was closed due to end-of-microbatch), so I wonder what causes "ERROR StreamingListenerBus: StreamingListenerBus has already stopped". Are you running in local-mode ("local[*]") ? or over YARN ? Any specific options