Re: Monitoring S3 Bucket with Spark Streaming

as a source here will let you process these notifications from your Spark streaming job. Nezih On Sat, Apr 9, 2016 at 10:58 AM Natu Lauchande <nlaucha...@gmail.com> wrote: > > Do you know if textFileStream can see if new files are created underneath > a whole bucket? >

Re: Monitoring S3 Bucket with Spark Streaming

While it is doable in Spark, S3 also supports notifications: http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html On Fri, Apr 8, 2016 at 9:15 PM Natu Lauchande wrote: > Hi Benjamin, > > I have done it . The critical configuration items are the ones below

Re: Amazon S3 Access Error

Did you take a look at this jira? On Wed, Apr 6, 2016 at 6:44 PM Joice Joy wrote: > I am facing an S3 access error when using Spark 1.6.1 pre-built for Hadoop > 2.6 or later. > But if I use Spark 1.6.1 pre-built for

Re: SparkContext.stop() takes too long to complete

ersion of hadoop do you use ? > > bq. Requesting to kill executor(s) 1136 > > Can you find more information on executor 1136 ? > > Thanks > > On Fri, Mar 18, 2016 at 4:16 PM, Nezih Yigitbasi < > nyigitb...@netflix.com.invalid> wrote: > >> Hi Spark experts

SparkContext.stop() takes too long to complete

Hi Spark experts, I am using Spark 1.5.2 on YARN with dynamic allocation enabled. I see in the driver/application master logs that the app is marked as SUCCEEDED and then SparkContext stop is called. However, this stop sequence takes > 10 minutes to complete, and YARN resource manager kills the

Re: question about combining small parquet files

teresting compaction approach of small files is discussed recently > > http://blog.cloudera.com/blog/2015/11/how-to-ingest-and-query-fast-data-with-impala-without-kudu/ > > > AFAIK Spark supports views too. > > > -- > Ruslan Dautkhanov > > On Thu, Nov 26, 201

question about combining small parquet files

the spark source). Any help is appreciated. Thanks, Nezih PS: this email is the same as my previous email as I learned that my previous email ended up as spam for many people since I sent it through nabble, sorry for the double post.

question about combining small input splits

it. Any help is appreciated. Thanks, Nezih -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/question-about-combining-small-input-splits-tp25440.html Sent from the Apache Spark User List mailing list archive at Nabble.com