Re: Observing DAGScheduler Log Messages

[M Bilal]

Thanks Jacek!

On Sun, Apr 7, 2019 at 8:21 PM Jacek Laskowski  wrote:

> Hi,
>
> Add the following line to conf/log4j.properties and you should have all
> the logs:
>
> log4j.logger.org.apache.spark.scheduler.DAGScheduler=ALL
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> Mastering Spark SQL https://bit.ly/mastering-spark-sql
> Spark Structured Streaming https://bit.ly/spark-structured-streaming
> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Sun, Apr 7, 2019 at 6:05 PM M Bilal  wrote:
>
>> Hi,
>>
>> I want to observe the log messages from DAGScheduler in Apache Spark.
>> Which log files do I need to check.
>> I have tried observing the driver logs and worker stderr logs but I can't
>> find any messages that are from that class.
>>
>>  I am using Spark 3.0.0 snapshot in standalone mode.
>>
>> Thanks.
>>
>> Regards,
>> Bilal
>>
>

Spark driver crashed with internal error

[Manu Zhang]

Hi all,

Recently, our Spark application's (2.3.1) driver has been crashing before
exiting with the following error.

* Could not load hsdis-amd64.so; library not loadable; PrintAssembly is
disabled*
* #*
* # A fatal error has been detected by the Java Runtime Environment:*
* #*
* #  Internal Error (sharedRuntime.cpp:834), pid=40111,
tid=0x2ac46180a700*
* #  fatal error: exception happened outside interpreter, nmethods and
vtable stubs at pc 0x2ac1a832edb1*
* #*
* # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build
1.8.0_131-b11)*
* # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode
linux-amd64 compressed oops)*

Similar errors were reported in older JVM and Linux  kernel versions
https://bugs.openjdk.java.net/browse/JDK-8203612 but we are running on Java
1.8.0_131 and kernel 3.10.0-693.21.1.el7.x86_64.

Here is the stack in error report.

*Current thread (0x2ac438005000):  JavaThread
"block-manager-slave-async-thread-pool-2" daemon [_thread_in_Java,
id=40362, stack(0x2ac46170a000,0x2ac46180b000)]*

*Stack: [0x2ac46170a000,0x2ac46180b000],  sp=0x2ac461808cd0,
free space=1019k*
*Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)*
*V  [libjvm.so+0xac826a]  VMError::report_and_die()+0x2ba*
*V  [libjvm.so+0x4fd089]  report_fatal(char const*, int, char const*)+0x59*
*V  [libjvm.so+0x9c391a]
SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned
char*, SharedRuntime::ImplicitExceptionKind)+0x33a*
*V  [libjvm.so+0x92bbfa]  JVM_handle_linux_signal+0x48a*
*V  [libjvm.so+0x921e13]  signalHandler(int, siginfo*, void*)+0x43*
*C  [libpthread.so.0+0xf5d0]*
*j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5*
*j  java.lang.Thread.run()V+11*
*v  ~StubRoutines::call_stub*
*V  [libjvm.so+0x691d16]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056*
*V  [libjvm.so+0x692221]  JavaCalls::call_virtual(JavaValue*, KlassHandle,
Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321*
*V  [libjvm.so+0x6926c7]  JavaCalls::call_virtual(JavaValue*, Handle,
KlassHandle, Symbol*, Symbol*, Thread*)+0x47*
*V  [libjvm.so+0x72da50]  thread_entry(JavaThread*, Thread*)+0xa0*
*V  [libjvm.so+0xa76833]  JavaThread::thread_main_inner()+0x103*
*V  [libjvm.so+0xa7697c]  JavaThread::run()+0x11c*
*V  [libjvm.so+0x927568]  java_start(Thread*)+0x108*
*C  [libpthread.so.0+0x7dd5]*

Has anyone seen this kind of error before ? I could provide more
information if needed.

Thanks,
Manu Zhang

Re: Observing DAGScheduler Log Messages

[Jacek Laskowski]

Hi,

Add the following line to conf/log4j.properties and you should have all the
logs:

log4j.logger.org.apache.spark.scheduler.DAGScheduler=ALL

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski


On Sun, Apr 7, 2019 at 6:05 PM M Bilal  wrote:

> Hi,
>
> I want to observe the log messages from DAGScheduler in Apache Spark.
> Which log files do I need to check.
> I have tried observing the driver logs and worker stderr logs but I can't
> find any messages that are from that class.
>
>  I am using Spark 3.0.0 snapshot in standalone mode.
>
> Thanks.
>
> Regards,
> Bilal
>

Observing DAGScheduler Log Messages

[M Bilal]

Hi,

I want to observe the log messages from DAGScheduler in Apache Spark. Which
log files do I need to check.
I have tried observing the driver logs and worker stderr logs but I can't
find any messages that are from that class.

 I am using Spark 3.0.0 snapshot in standalone mode.

Thanks.

Regards,
Bilal

Re: Is there any spark API function to handle a group of companies at once in this scenario?

[Mich Talebzadeh]

Are these ticker prices for these companies like share value etc?

How do you get this company data in Spark? Are you using Spark streaming to
get the prices, then work out the stats (AVG, STDDEV etc) and put them back
into DB?

HTH

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 5 Apr 2019 at 10:51, Shyam P  wrote:

> Hi ,
> In my scenario I have few companies , for which I need to calculate few
> stats like avg I need to be stored in Cassandra , for next set of records I
> need to get previously calculated and over it i need to calculate
> accumulated results ( i.e preset set of data + previously stored stats) and
> stored it back to Cassandra.
>
> what function/API of spark be used while calculating the above for a group
> of companies?
>
>
> Regards,
> Shyam
>