Yes Mich,
Mapping the spark sql query that got executed corresponding to an
application Id on yarn would greatly help in analyzing and debugging the
query for any potential problems.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com
Mich,
The directory is already there and event logs are getting generated, I have
checked them it contains the query plan but not the actual query.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com
Yes i can see the jobs on 8088 and also on the spark history url. spark
history server is showing up the plan details on the sql tab but not giving
the query.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com
Hi Mich,
I do see the .scala_history directory, but it contains all the queries which
got executed uptill now, but if i have to map a specific query to an
application Id in yarn that would not correlate, hence this method alone
won't suffice
Thanks,
Arpan Bhandari
--
Sent from:
Hi Mich,
Repeated the steps as suggested, but still there is no such folder created
in the home directory. Do we need to enable some property so that it creates
one.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com
Sanchit,
It seems I have to do some sort of analysis from the plan to get the query.
Appreciate all your help on this.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To
Sachit,
That is showing all the queries that got executed, but how it would get
mapped to specific application Id it was associated with ?
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com
Hey Mich,
Thanks for the suggestions, but i don't see any such folder created on the
edge node.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To unsubscribe e-mail:
Hey Sachit,
It shows the query plan, which is difficult to diagnose out and depict the
actual query.
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To unsubscribe e-mail: user
Hi Sachit,
Yes it was executed using spark shell, history is already enabled. already
checked sql tab but it is not showing the query. My spark version is 2.4.5
Thanks,
Arpan Bhandari
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com
Hi ,
Is there a way to track back spark sql after it has been already run i.e.
query has been already submitted by a person and i have to back trace what
query actually got submitted.
Appreciate any help on this.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
11 matches
Mail list logo
