Spark REST API YARN client mode is not full?

2016-10-06 Thread Vladimir Tretyakov
Hi, When I start Spark v1.6 (cdh5.8.0) in Yarn client mode I see that 4040 port is avaiable, but UI shows nothing and API returns not full information. I started Spark application like this: spark-submit --master yarn-client --class org.apache.spark.examples.SparkPi

Re: Spark metrics when running with YARN?

2016-09-18 Thread Vladimir Tretyakov
could try running simple Spark applications > on Standalone and Yarn. > > On Fri, Sep 16, 2016 at 10:32 PM, Vladimir Tretyakov < > vladimir.tretya...@sematext.com> wrote: > >> Hello. >> >> Found that there is also Spark metric Sink like MetricsServlet. >> which is e

Re: Spark metrics when running with YARN?

2016-09-16 Thread Vladimir Tretyakov
other than Standalone? Why are there 2 ways to get information, REST API and this Sink? Best regards, Vladimir. On Mon, Sep 12, 2016 at 3:53 PM, Vladimir Tretyakov < vladimir.tretya...@sematext.com> wrote: > Hello Saisai Shao, Jacek Laskowski , thx for information. > >

Re: Spark metrics when running with YARN?

2016-09-12 Thread Vladimir Tretyakov
https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> >> >> On Sun, Sep 11, 2016 at 11:18 AM, Vladimir Tretyakov >> <vladimir.tretya...@sematext.com> wrote:

Re: Spark metrics when running with YARN?

2016-09-11 Thread Vladimir Tretyakov
> Hi, > > That's correct. One app one web UI. Open 4041 and you'll see the other > app. > > Jacek > > On 9 Sep 2016 11:53 a.m., "Vladimir Tretyakov" < > vladimir.tretya...@sematext.com> wrote: > >> Hello again. >> >> I am trying to pl

Is there a way to provide individual property to each Spark executor?

2014-10-02 Thread Vladimir Tretyakov
/spm-monitor-spark.jar=myValue-*2* :spark-executor:default ... ... ... Can I do something like that in Spark for executor? If not maybe it can be done in the future? Will be useful. Thx, best redgards, Vladimir Tretyakov.

JMXSink for YARN deployment

2014-09-11 Thread Vladimir Tretyakov
Hello, we are in Sematext (https://apps.sematext.com/) are writing Monitoring tool for Spark and we came across one question: How to enable JMX metrics for YARN deployment? We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink to file $SPARK_HOME/conf/metrics.properties but it doesn't

Re: JMXSink for YARN deployment

2014-09-11 Thread Vladimir Tretyakov
in the yarn container, so metrics system cannot load the right sinks. Thanks Jerry *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] *Sent:* Thursday, September 11, 2014 7:30 PM *To:* user@spark.apache.org *Subject:* JMXSink for YARN deployment Hello, we are in Sematext

Re: JMXSink for YARN deployment

2014-09-11 Thread Vladimir Tretyakov
should find this file in their local FS because this file is loaded locally. Besides I think this might be a kind of workaround, a better solution is to fix this by some other solutions. Thanks Jerry *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] *Sent

Re: JMXSink for YARN deployment

2014-09-11 Thread Vladimir Tretyakov
. On Thu, Sep 11, 2014 at 5:55 PM, Kousuke Saruta saru...@oss.nttdata.co.jp wrote: Hi Vladimir How about use --files option with spark-submit? - Kousuke (2014/09/11 23:43), Vladimir Tretyakov wrote: Hi again, yeah , I've tried to use ” spark.metrics.conf” before my question in ML