Re: Spark job profiler results showing high TCP cpu time

2017-06-28 Thread Reth RM 
I am using visual vm:  https://github.com/krasa/VisualVMLauncher

@Marcelo, thank you for the reply, that was helpful.


On Fri, Jun 23, 2017 at 12:48 PM, Eduardo Mello 
wrote:

> what program do u use to profile Spark?
>
> On Fri, Jun 23, 2017 at 3:07 PM, Marcelo Vanzin 
> wrote:
>
>> That thread looks like the connection between the Spark process and
>> jvisualvm. It's expected to show high up when doing sampling if the
>> app is not doing much else.
>>
>> On Fri, Jun 23, 2017 at 10:46 AM, Reth RM  wrote:
>> > Running a spark job on local machine and profiler results indicate that
>> > highest time spent in sun.rmi.transport.tcp.TCPTrans
>> port$ConnectionHandler.
>> > Screenshot of profiler result can be seen here : https://jpst.it/10i-V
>> >
>> > Spark job(program) is performing IO (sc.wholeTextFile method of spark
>> apis),
>> > Reads files from local file system and analyses the text to obtain
>> tokens.
>> >
>> > Any thoughts and suggestions?
>> >
>> > Thanks.
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>

Re: Spark job profiler results showing high TCP cpu time

2017-06-23 Thread Eduardo Mello 
what program do u use to profile Spark?

On Fri, Jun 23, 2017 at 3:07 PM, Marcelo Vanzin  wrote:

> That thread looks like the connection between the Spark process and
> jvisualvm. It's expected to show high up when doing sampling if the
> app is not doing much else.
>
> On Fri, Jun 23, 2017 at 10:46 AM, Reth RM  wrote:
> > Running a spark job on local machine and profiler results indicate that
> > highest time spent in sun.rmi.transport.tcp.TCPTransport$
> ConnectionHandler.
> > Screenshot of profiler result can be seen here : https://jpst.it/10i-V
> >
> > Spark job(program) is performing IO (sc.wholeTextFile method of spark
> apis),
> > Reads files from local file system and analyses the text to obtain
> tokens.
> >
> > Any thoughts and suggestions?
> >
> > Thanks.
> >
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: Spark job profiler results showing high TCP cpu time

2017-06-23 Thread Marcelo Vanzin 
That thread looks like the connection between the Spark process and
jvisualvm. It's expected to show high up when doing sampling if the
app is not doing much else.

On Fri, Jun 23, 2017 at 10:46 AM, Reth RM  wrote:
> Running a spark job on local machine and profiler results indicate that
> highest time spent in sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.
> Screenshot of profiler result can be seen here : https://jpst.it/10i-V
>
> Spark job(program) is performing IO (sc.wholeTextFile method of spark apis),
> Reads files from local file system and analyses the text to obtain tokens.
>
> Any thoughts and suggestions?
>
> Thanks.
>



-- 
Marcelo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark job profiler results showing high TCP cpu time

2017-06-23 Thread Reth RM 
Running a spark job on local machine and profiler results indicate that
highest time spent in
*sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.* Screenshot of
profiler result can be seen here : https://jpst.it/10i-V

Spark job(program) is performing IO (sc.wholeTextFile method of spark
apis), Reads files from local file system and analyses the text to obtain
tokens.

Any thoughts and suggestions?

Thanks.