Re: Spark shell never leaves ACCEPTED state in YARN CDH5

2015-03-25 Thread Dean Chen
of the CLI node. -- Dean Chen On Wed, Mar 25, 2015 at 12:18 PM, Marcelo Vanzin van...@cloudera.com wrote: The probably means there are not enough free resources in your cluster to run the AM for the Spark job. Check your RM's web ui to see the resources you have available. On Wed, Mar 25, 2015

Spark Stream + HDFS Append

2014-08-24 Thread Dean Chen
We are using HDFS for log storage where logs are flushed to HDFS every minute, with a new file created for each hour. We would like to consume these logs using spark streaming.  The docs state that new HDFS will be picked up, but does Spark Streaming support HDFS appends? — Dean Chen