Re: Structured Streaming and Microbatches

2016-07-13 Thread Jacek Laskowski
Hi, It's still microbatching architecture with triggers as batchIntervals. It's just faster by default and the API is more pleasant, i.e. Dataset-driven. Jacek On 13 Jul 2016 10:35 p.m., "Matthias Niehoff" < matthias.nieh...@codecentric.de> wrote: Hi everybody, as far as I understand with new t

Structured Streaming and Microbatches

2016-07-13 Thread Matthias Niehoff
Hi everybody, as far as I understand with new the structured Streaming API the output will not get processed every x seconds anymore. Instead the data will be processed as soon as is arrived. But there might be a delay due to processing time for the data. A small example: Data comes in and the pr

Re: Spark Homepage

2016-07-13 Thread Dongjoon Hyun
Thank you for the pointer, Reynold! I see the situation now. On Wed, Jul 13, 2016 at 12:09 PM, Reynold Xin wrote: > It's related to > https://issues.apache.org/jira/servicedesk/agent/INFRA/issue/INFRA-12055 > > > > On Wed, Jul 13, 2016 at 12:07 PM, Holden Karau > wrote: > >> This has also been

Re: Spark Homepage

2016-07-13 Thread Reynold Xin
It's related to https://issues.apache.org/jira/servicedesk/agent/INFRA/issue/INFRA-12055 On Wed, Jul 13, 2016 at 12:07 PM, Holden Karau wrote: > This has also been reported on the user@ by a few people - other apache > projects (arrow & hadoop) don't seem to be affected so maybe it was a just

Re: Spark Homepage

2016-07-13 Thread Dongjoon Hyun
Oh, thank you, Holden. Next time, I'll search user@ first before asking here. :) Dongjoon On Wed, Jul 13, 2016 at 12:07 PM, Holden Karau wrote: > This has also been reported on the user@ by a few people - other apache > projects (arrow & hadoop) don't seem to be affected so maybe it was a just

Re: Spark Homepage

2016-07-13 Thread Holden Karau
This has also been reported on the user@ by a few people - other apache projects (arrow & hadoop) don't seem to be affected so maybe it was a just bad update for the Spark website? On Wed, Jul 13, 2016 at 12:05 PM, Dongjoon Hyun wrote: > Hi, All. > > Currently, Spark Homepage (http://spark.apach

Spark Homepage

2016-07-13 Thread Dongjoon Hyun
Hi, All. Currently, Spark Homepage (http://spark.apache.org/) shows file listing (containing md files) Is there any maintenance operation on that? :) Warmly, Dongjoon.

Re: When worker is killed driver continues to run causing issues in supervise mode

2016-07-13 Thread Noorul Islam Kamal Malmiyoda
Adding dev list On Jul 13, 2016 5:38 PM, "Noorul Islam K M" wrote: > > Spark version: 1.6.1 > Cluster Manager: Standalone > > I am experimenting with cluster mode deployment along with supervise for > high availability of streaming applications. > > 1. Submit a streaming job in cluster mode with

Re: Why isnt spark-yarn module is excluded from the spark parent pom?

2016-07-13 Thread Jacek Laskowski
And the reason is that not all Spark installations are for YARN as the cluster manager. Jacek On 13 Jul 2016 9:23 a.m., "Sean Owen" wrote: > It's activated by a profile called 'yarn', like several other modules. > > On Wed, Jul 13, 2016 at 5:15 AM, Niranda Perera > wrote: > > Hi guys, > > > > I

Re: Why isnt spark-yarn module is excluded from the spark parent pom?

2016-07-13 Thread Sean Owen
It's activated by a profile called 'yarn', like several other modules. On Wed, Jul 13, 2016 at 5:15 AM, Niranda Perera wrote: > Hi guys, > > I could not find the spark-yarn module in the spark parent pom? Is there any > particular reason why this has been excluded? > > Best > -- > Niranda > @n1r4