Re: Spark History Server NOT showing Jobs with Hortonworks

2016-02-19 Thread Steve Loughran 

this is set up to save history to the timeline service, something which works 
provided the applications are all set up to publish there too.

On 18 Feb 2016, at 22:22, Sutanu Das mailto:sd2...@att.com>> 
wrote:

Hi Community,

Challenged with Spark issues with Hortonworks  (HDP 2.3.2_Spark 1.4.1) – The 
Spark History Server is NOT showing the Spark Running Jobs in Local Mode

The local-host:4040/app/v1 is ALSO not working

How can I look at my local Spark job?


# Generated by Apache Ambari. Fri Feb  5 00:37:06 2016

spark.history.kerberos.keytab none
spark.history.kerberos.principal none


this tells the history server to use ATS
spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider


spark.history.ui.port 18080
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 2048
spark.yarn.executor.memoryOverhead 2048
spark.yarn.historyServer.address 
has-dal-0001.corp.wayport.net:18080
spark.yarn.max.executor.failures 3
spark.yarn.preserve.staging.files false
spark.yarn.queue default
spark.yarn.scheduler.heartbeat.interval-ms 5000

this says: publish via it
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.yarn.submit.file.replication 3



There's some asynchronous publishing, so things don't appear immediately as the 
app starts, and the updates can take a bit to trickle out, but things look set 
up right to work both ways.


I'll email you off the list and see if I can help track down what's happening

-steve

Re: Spark History Server NOT showing Jobs with Hortonworks

2016-02-18 Thread Divya Gehlot 
Hi Sutanu ,

When you run your spark shell
you would  see below lines in your console

16/02/18 21:43:53 INFO AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4041
16/02/18 21:43:53 INFO Utils: Successfully started service 'SparkUI' on
port 4041.
16/02/18 21:43:54 INFO SparkUI: Started SparkUI at http://xx.xx.xx.xxx:4041

As In my case instead of default port the UI started at 4041 port .

Hope this helps.

Thanks,
Divya



On 19 February 2016 at 07:09, Mich Talebzadeh  wrote:

> Is 4040 port used in your host? It should be default
>
>
>
> Example
>
>
>
> *netstat -plten|grep 4040*
>
>
>
> tcp0  0 :::4040
> :::*LISTEN  1009   42748209   *22778*/java
>
>
>
> *ps -ef|grep 22778*
>
>
>
> hduser   22778 22770  0 08:34 pts/100:01:18 /usr/java/latest/bin/java
> -cp
> /home/hduser/jars/jconn4.jar:/home/hduser/jars/ojdbc6.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/conf/:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/hadoop-2.6.0/etc/hadoop/
> -Dscala.usejavacp=true -Xms1G -Xmx1G -XX:MaxPermSize=256m
> org.apache.spark.deploy.SparkSubmit --master spark://50.140.197.217:7077
> --class org.apache.spark.repl.Main --name Spark shell spark-shell
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
> *From:* Sutanu Das [mailto:sd2...@att.com]
> *Sent:* 18 February 2016 22:58
> *To:* Mich Talebzadeh ; user@spark.apache.org
>
> *Subject:* RE: Spark History Server NOT showing Jobs with Hortonworks
>
>
>
> Hi Mich, Community - Do I need to specify it in the properties file in my
> spark-submit ?
>
>
>
> *From:* Mich Talebzadeh [mailto:m...@peridale.co.uk ]
>
> *Sent:* Thursday, February 18, 2016 4:28 PM
> *To:* Sutanu Das; user@spark.apache.org
> *Subject:* RE: Spark History Server NOT showing Jobs with Hortonworks
>
>
>
> The jobs are normally shown under :4040/jobs/ in a normal set up
> not using any vendor’s flavoiur
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
> *From:* Sutanu Das [mailto:sd2...@att.com ]
> *Sent:* 18 February 2016 22:22
> *To:* user@spark.apache.org
> *Subject:* Spark History Server NOT showing Jobs with Hortonworks
>
>
>
> Hi Community,
>
>
>
> Challenged with Spark issues with *Hortonworks*  (HDP 2.3.2_Spark 1.4.1)
> – The Spark History Server is NOT showing the Spark Running Jobs in Local
> Mode
>
>
>
> The local-host:4040/app/v1 is ALSO not working
>
>
>
> How can I look at my local Spark job?
>
>
>
>
>
> # Generated by Apache Ambari. Fri Feb  5 00:37:06 2016
>
>
>
> spark.history.kerberos.keytab none
>
> spark.history.kerberos.principal none
>
> spark.history.provider
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>
> spark.history.ui.port 18080
>
> spark.yarn

RE: Spark History Server NOT showing Jobs with Hortonworks

2016-02-18 Thread Mich Talebzadeh 
Is 4040 port used in your host? It should be default

 

Example

 

netstat -plten|grep 4040

 

tcp0  0 :::4040 :::*
LISTEN  1009   42748209   22778/java

 

ps -ef|grep 22778

 

hduser   22778 22770  0 08:34 pts/100:01:18 /usr/java/latest/bin/java
-cp
/home/hduser/jars/jconn4.jar:/home/hduser/jars/ojdbc6.jar:/usr/lib/spark-1.5
.2-bin-hadoop2.6/conf/:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/spark-assembly
-1.5.2-hadoop2.6.0.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-co
re-3.2.10.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2
.6.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/h
ome/hduser/hadoop-2.6.0/etc/hadoop/ -Dscala.usejavacp=true -Xms1G -Xmx1G
-XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --master
spark://50.140.197.217:7077 --class org.apache.spark.repl.Main --name Spark
shell spark-shell

 

Dr Mich Talebzadeh

 

LinkedIn
<https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABU
rV8Pw>
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUr
V8Pw

 

 <http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Technology Ltd, its subsidiaries nor their
employees accept any responsibility.

 

 

From: Sutanu Das [mailto:sd2...@att.com] 
Sent: 18 February 2016 22:58
To: Mich Talebzadeh ; user@spark.apache.org
Subject: RE: Spark History Server NOT showing Jobs with Hortonworks

 

Hi Mich, Community - Do I need to specify it in the properties file in my
spark-submit ?

 

From: Mich Talebzadeh [mailto:m...@peridale.co.uk] 
Sent: Thursday, February 18, 2016 4:28 PM
To: Sutanu Das; user@spark.apache.org <mailto:user@spark.apache.org> 
Subject: RE: Spark History Server NOT showing Jobs with Hortonworks

 

The jobs are normally shown under :4040/jobs/ in a normal set up
not using any vendor's flavoiur

 

Dr Mich Talebzadeh

 

LinkedIn
<https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABU
rV8Pw>
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUr
V8Pw

 

 <http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Technology Ltd, its subsidiaries nor their
employees accept any responsibility.

 

 

From: Sutanu Das [mailto:sd2...@att.com] 
Sent: 18 February 2016 22:22
To: user@spark.apache.org <mailto:user@spark.apache.org> 
Subject: Spark History Server NOT showing Jobs with Hortonworks

 

Hi Community,

 

Challenged with Spark issues with Hortonworks  (HDP 2.3.2_Spark 1.4.1) - The
Spark History Server is NOT showing the Spark Running Jobs in Local Mode 

 

The local-host:4040/app/v1 is ALSO not working

 

How can I look at my local Spark job?

 

 

# Generated by Apache Ambari. Fri Feb  5 00:37:06 2016



spark.history.kerberos.keytab none

spark.history.kerberos.principal none

spark.history.provider
org.apache.spark.deploy.yarn.history.YarnHistoryProvider

spark.history.ui.port 18080

spark.yarn.containerLauncherMaxThreads 25

spark.yarn.driver.memoryOverhead 2048

spark.yarn.executor.memoryOverhead 2048

spark.yarn.historyServer.address has-dal-0001.corp.wayport.net:18080

spark.yarn.max.executor.failures 3

spark.yarn.preserve.staging.files false

spark.yarn.queue default

spark.yarn.scheduler.heartbeat.interval-ms 5000

spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

spark.yarn.submit.file.replication 3

 

History Server 

*   Timeline Service Location:
http://has-dal-0002.corp.wayport.net:8188/
*   Last Updated: Feb 18, 2016 10:09:12 PM UTC
*   Service Started: Feb 5, 2016 12:37:15 AM UTC
*   Current Time: Feb 18, 2016 10:10:46 PM UTC
*   Timeline Service: Timeline service is enabled
*   History Provider: Apache Hadoop YARN Timeline Service

RE: Spark History Server NOT showing Jobs with Hortonworks

2016-02-18 Thread Sutanu Das 
Hi Mich, Community - Do I need to specify it in the properties file in my 
spark-submit ?

From: Mich Talebzadeh [mailto:m...@peridale.co.uk]
Sent: Thursday, February 18, 2016 4:28 PM
To: Sutanu Das; user@spark.apache.org
Subject: RE: Spark History Server NOT showing Jobs with Hortonworks

The jobs are normally shown under :4040/jobs/ in a normal set up not 
using any vendor's flavoiur

Dr Mich Talebzadeh

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>

NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by Peridale Technology Ltd, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither Peridale Technology Ltd, its subsidiaries nor their employees 
accept any responsibility.


From: Sutanu Das [mailto:sd2...@att.com]
Sent: 18 February 2016 22:22
To: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Spark History Server NOT showing Jobs with Hortonworks

Hi Community,

Challenged with Spark issues with Hortonworks  (HDP 2.3.2_Spark 1.4.1) - The 
Spark History Server is NOT showing the Spark Running Jobs in Local Mode

The local-host:4040/app/v1 is ALSO not working

How can I look at my local Spark job?


# Generated by Apache Ambari. Fri Feb  5 00:37:06 2016

spark.history.kerberos.keytab none
spark.history.kerberos.principal none
spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.history.ui.port 18080
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 2048
spark.yarn.executor.memoryOverhead 2048
spark.yarn.historyServer.address has-dal-0001.corp.wayport.net:18080
spark.yarn.max.executor.failures 3
spark.yarn.preserve.staging.files false
spark.yarn.queue default
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.yarn.submit.file.replication 3

History Server

  *   Timeline Service Location: http://has-dal-0002.corp.wayport.net:8188/
  *   Last Updated: Feb 18, 2016 10:09:12 PM UTC
  *   Service Started: Feb 5, 2016 12:37:15 AM UTC
  *   Current Time: Feb 18, 2016 10:10:46 PM UTC
  *   Timeline Service: Timeline service is enabled
  *   History Provider: Apache Hadoop YARN Timeline Service

RE: Spark History Server NOT showing Jobs with Hortonworks

2016-02-18 Thread Mich Talebzadeh 
The jobs are normally shown under :4040/jobs/ in a normal set up
not using any vendor's flavoiur

 

Dr Mich Talebzadeh

 

LinkedIn

https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUr
V8Pw

 

  http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Technology Ltd, its subsidiaries nor their
employees accept any responsibility.

 

 

From: Sutanu Das [mailto:sd2...@att.com] 
Sent: 18 February 2016 22:22
To: user@spark.apache.org
Subject: Spark History Server NOT showing Jobs with Hortonworks

 

Hi Community,

 

Challenged with Spark issues with Hortonworks  (HDP 2.3.2_Spark 1.4.1) - The
Spark History Server is NOT showing the Spark Running Jobs in Local Mode 

 

The local-host:4040/app/v1 is ALSO not working

 

How can I look at my local Spark job?

 

 

# Generated by Apache Ambari. Fri Feb  5 00:37:06 2016



spark.history.kerberos.keytab none

spark.history.kerberos.principal none

spark.history.provider
org.apache.spark.deploy.yarn.history.YarnHistoryProvider

spark.history.ui.port 18080

spark.yarn.containerLauncherMaxThreads 25

spark.yarn.driver.memoryOverhead 2048

spark.yarn.executor.memoryOverhead 2048

spark.yarn.historyServer.address has-dal-0001.corp.wayport.net:18080

spark.yarn.max.executor.failures 3

spark.yarn.preserve.staging.files false

spark.yarn.queue default

spark.yarn.scheduler.heartbeat.interval-ms 5000

spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

spark.yarn.submit.file.replication 3

 

History Server 

*   Timeline Service Location:
http://has-dal-0002.corp.wayport.net:8188/
*   Last Updated: Feb 18, 2016 10:09:12 PM UTC
*   Service Started: Feb 5, 2016 12:37:15 AM UTC
*   Current Time: Feb 18, 2016 10:10:46 PM UTC
*   Timeline Service: Timeline service is enabled
*   History Provider: Apache Hadoop YARN Timeline Service