[jira] [Commented] (SPARK-34674) Spark app on k8s doesn't terminate without call to sparkContext.stop() method
[
https://issues.apache.org/jira/browse/SPARK-34674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303313#comment-17303313
]
Sergey commented on SPARK-34674:
Sorry, I didn't search sufficiently for existing issues.
There is already the issue https://issues.apache.org/jira/browse/SPARK-27812,
and it is marked as Fixed.
But it looks like, I have this bug in Spark 3.1.1.
I just start an spark app on Amazon EKS (Kubernetes version - 1.17) by
_spark-on-k8s-operator: v1beta2-1.2.0-3.0.0_
[(https://github.com/GoogleCloudPlatform/spark-on-k8s-operator)|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator]
Spark docker image is built from the official release of spark-3.1.1 hadoop3.2.
> Spark app on k8s doesn't terminate without call to sparkContext.stop() method
> -
>
> Key: SPARK-34674
> URL: https://issues.apache.org/jira/browse/SPARK-34674
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.1
>Reporter: Sergey
>Priority: Major
>
> Hello!
> I have run into a problem that if I don't call the method
> sparkContext.stop() explicitly, then a Spark driver process doesn't terminate
> even after its Main method has been completed. This behaviour is different
> from spark on yarn, where the manual sparkContext stopping is not required.
> It looks like, the problem is in using non-daemon threads, which prevent the
> driver jvm process from terminating.
> At least I see two non-daemon threads, if I don't call sparkContext.stop():
> {code:java}
> Thread[OkHttp kubernetes.default.svc,5,main]
> Thread[OkHttp kubernetes.default.svc Writer,5,main]
> {code}
> Could you tell please, if it is possible to solve this problem?
> Docker image from the official release of spark-3.1.1 hadoop3.2 is used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.
[
https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303304#comment-17303304
]
Sergey edited comment on SPARK-27812 at 3/17/21, 11:18 AM:
---
It seems, I have reproduced this bug in Spark 3.1.1.
If I don't call sparkContext.stop() explicitly, then a Spark driver process
doesn't terminate even after its Main method has been completed.
There are two non-daemon threads, if I don't call sparkContext.stop():
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]{code}
It looks like, it prevents the driver jvm process from terminating.
Spark app is started on Amazon EKS (Kubernetes version - 1.17) by
_spark-on-k8s-operator: v1beta2-1.2.0-3.0.0_
[(https://github.com/GoogleCloudPlatform/spark-on-k8s-operator)|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator]
Spark docker image is built from the official release of spark-3.1.1 hadoop3.2.
was (Author: kotlov):
It seems, I have reproduced this bug in Spark 3.1.1.
If I don't call sparkContext.stop() explicitly, then a Spark driver process
doesn't terminate even after its Main method has been completed.
There are two non-daemon threads, if I don't call sparkContext.stop():
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]{code}
It looks like, it prevents the driver jvm process from terminating.
Spark app is started on Amazon EKS (Kubernetes version - 1.17) by
[spark-on-k8s-operator:
v1beta2-1.2.0-3.0.0|[https://github.com/GoogleCloudPlatform/spark-on-k8s-operator].]
Spark docker image is built from the official release of spark-3.1.1 hadoop3.2.
> kubernetes client import non-daemon thread which block jvm exit.
>
>
> Key: SPARK-27812
> URL: https://issues.apache.org/jira/browse/SPARK-27812
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Core
>Affects Versions: 2.4.3, 2.4.4
>Reporter: Henry Yu
>Assignee: Igor Calabria
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
>
> I try spark-submit to k8s with cluster mode. Driver pod failed to exit with
> An Okhttp Websocket Non-Daemon Thread.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.
[
https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303304#comment-17303304
]
Sergey commented on SPARK-27812:
It seems, I have reproduced this bug in Spark 3.1.1.
If I don't call sparkContext.stop() explicitly, then a Spark driver process
doesn't terminate even after its Main method has been completed.
There are two non-daemon threads, if I don't call sparkContext.stop():
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]{code}
It looks like, it prevents the driver jvm process from terminating.
Spark app is started on Amazon EKS (Kubernetes version - 1.17) by
[spark-on-k8s-operator:
v1beta2-1.2.0-3.0.0|[https://github.com/GoogleCloudPlatform/spark-on-k8s-operator].]
Spark docker image is built from the official release of spark-3.1.1 hadoop3.2.
> kubernetes client import non-daemon thread which block jvm exit.
>
>
> Key: SPARK-27812
> URL: https://issues.apache.org/jira/browse/SPARK-27812
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Core
>Affects Versions: 2.4.3, 2.4.4
>Reporter: Henry Yu
>Assignee: Igor Calabria
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
>
> I try spark-submit to k8s with cluster mode. Driver pod failed to exit with
> An Okhttp Websocket Non-Daemon Thread.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-34674) Spark app on k8s doesn't terminate without call to sparkContext.stop() method
Sergey created SPARK-34674:
--
Summary: Spark app on k8s doesn't terminate without call to
sparkContext.stop() method
Key: SPARK-34674
URL: https://issues.apache.org/jira/browse/SPARK-34674
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.1.1
Reporter: Sergey
Hello!
I have run into a problem that if I don't call the method sparkContext.stop()
explicitly, then a Spark driver process doesn't terminate even after its Main
method has been completed. This behaviour is different from spark on yarn,
where the manual sparkContext stopping is not required.
It looks like, the problem is in using non-daemon threads, which prevent the
driver jvm process from terminating.
At least I see two non-daemon threads, if I don't call sparkContext.stop():
{{ }}
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]
{code}
{{}}
Could you tell please, if it is possible to solve this problem?
Docker image from the official release of spark-3.1.1 hadoop3.2 is used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34674) Spark app on k8s doesn't terminate without call to sparkContext.stop() method
[
https://issues.apache.org/jira/browse/SPARK-34674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-34674:
---
Description:
Hello!
I have run into a problem that if I don't call the method sparkContext.stop()
explicitly, then a Spark driver process doesn't terminate even after its Main
method has been completed. This behaviour is different from spark on yarn,
where the manual sparkContext stopping is not required.
It looks like, the problem is in using non-daemon threads, which prevent the
driver jvm process from terminating.
At least I see two non-daemon threads, if I don't call sparkContext.stop():
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]
{code}
Could you tell please, if it is possible to solve this problem?
Docker image from the official release of spark-3.1.1 hadoop3.2 is used.
was:
Hello!
I have run into a problem that if I don't call the method sparkContext.stop()
explicitly, then a Spark driver process doesn't terminate even after its Main
method has been completed. This behaviour is different from spark on yarn,
where the manual sparkContext stopping is not required.
It looks like, the problem is in using non-daemon threads, which prevent the
driver jvm process from terminating.
At least I see two non-daemon threads, if I don't call sparkContext.stop():
{{ }}
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]
{code}
{{}}
Could you tell please, if it is possible to solve this problem?
Docker image from the official release of spark-3.1.1 hadoop3.2 is used.
> Spark app on k8s doesn't terminate without call to sparkContext.stop() method
> -
>
> Key: SPARK-34674
> URL: https://issues.apache.org/jira/browse/SPARK-34674
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.1
>Reporter: Sergey
>Priority: Major
>
> Hello!
> I have run into a problem that if I don't call the method
> sparkContext.stop() explicitly, then a Spark driver process doesn't terminate
> even after its Main method has been completed. This behaviour is different
> from spark on yarn, where the manual sparkContext stopping is not required.
> It looks like, the problem is in using non-daemon threads, which prevent the
> driver jvm process from terminating.
> At least I see two non-daemon threads, if I don't call sparkContext.stop():
> {code:java}
> Thread[OkHttp kubernetes.default.svc,5,main]
> Thread[OkHttp kubernetes.default.svc Writer,5,main]
> {code}
> Could you tell please, if it is possible to solve this problem?
> Docker image from the official release of spark-3.1.1 hadoop3.2 is used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-26688: --- Comment: was deleted (was: Hi Imran, thanks for you reply. "Meanwhile devs start to apply this willy-nilly, as these configs tend to just keep getting built up over time " SLA could mitigate this problem. Every blacklisted node for a specific job slows it down in a long run. Anyway, dev would have to report / communicate with ops to resolve node issue. "Ideally, blacklisting and speculation should be able to prevent that problem" We are going to try out speculation but we are not there yet. ) > Provide configuration of initially blacklisted YARN nodes > - > > Key: SPARK-26688 > URL: https://issues.apache.org/jira/browse/SPARK-26688 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.0.0 >Reporter: Attila Zsolt Piros >Assignee: Attila Zsolt Piros >Priority: Major > Fix For: 3.0.0 > > > Introducing new config for initially blacklisted YARN nodes. > This came up in the apache spark user mailing list: > [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-26688: --- Comment: was deleted (was: Hi There! I'm very glad that the community paid attention to my question. Let me try to explain usecase There is 1K nodes cluster and jobs have performance degradation because of a single node. It's rather hard to convince Cluster Ops to decommission node because of "performance degradation". Imagine 10 dev teams chase single ops team for valid reason (node has problems) or because code has a bug or data is skewed or spots on the sun. We can't just decommission node because random dev complains. Simple solution: * rerun failed / delayed job and blacklist "problematic" node in advance. * Report about the problem if job works w/o anomalies. * ops collect complains about node and start to decommission it when "complains threshold" is reached. It's a rather low probability that many loosely coupled teams with loosely coupled jobs complain about a single node. Results * Ops are not spammed with a random requests from devs * devs are not blocked because of the really bad node. * it's very cheap for everyone to "blacklist" node during job submission w/o doing anything to node. ) > Provide configuration of initially blacklisted YARN nodes > - > > Key: SPARK-26688 > URL: https://issues.apache.org/jira/browse/SPARK-26688 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.0.0 >Reporter: Attila Zsolt Piros >Assignee: Attila Zsolt Piros >Priority: Major > Fix For: 3.0.0 > > > Introducing new config for initially blacklisted YARN nodes. > This came up in the apache spark user mailing list: > [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27543) Support getRequiredJars and getRequiredFiles APIs for Hive UDFs
[ https://issues.apache.org/jira/browse/SPARK-27543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-27543: --- Issue Type: Improvement (was: Bug) > Support getRequiredJars and getRequiredFiles APIs for Hive UDFs > --- > > Key: SPARK-27543 > URL: https://issues.apache.org/jira/browse/SPARK-27543 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.0.0, 2.4.1 >Reporter: Sergey >Priority: Minor > Original Estimate: 1,344h > Remaining Estimate: 1,344h > > *getRequiredJars* and *getRequiredFiles* - functions to automatically include > additional resources required by a UDF. The files that are provided in > methods would be accessible by executors by simple file name. This is > necessary for UDFs that need to have some required files distributed, or > classes from third-party jars to be available from executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27543) Support getRequiredJars and getRequiredFiles APIs for Hive UDFs
Sergey created SPARK-27543: -- Summary: Support getRequiredJars and getRequiredFiles APIs for Hive UDFs Key: SPARK-27543 URL: https://issues.apache.org/jira/browse/SPARK-27543 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.1, 2.0.0 Reporter: Sergey *getRequiredJars* and *getRequiredFiles* - functions to automatically include additional resources required by a UDF. The files that are provided in methods would be accessible by executors by simple file name. This is necessary for UDFs that need to have some required files distributed, or classes from third-party jars to be available from executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes
[
https://issues.apache.org/jira/browse/SPARK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-23773:
---
Issue Type: Bug (was: Improvement)
> JacksonGenerator does not include keys that have null value for StructTypes
> ---
>
> Key: SPARK-23773
> URL: https://issues.apache.org/jira/browse/SPARK-23773
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0, 2.3.0
>Reporter: Sergey
>Priority: Trivial
>
> When "toJSON" is called on a dataset, the result JSON string will not have
> keys displayed for StructTypes that have null value.
> Repro:
> {noformat}
> scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> ...
> scala> val df = sqlContext.sql(""" select NAMED_STRUCT('f1', null, 'f2',
> ARRAY(TRUE, FALSE), 'f3', MAP(123L, 123.456), 'f4', 'some string') as
> my_struct """)
> ...
> scala> df.toJSON.collect().foreach(println)
> {"my_struct":{"f2":[true,false],"f3":({"123":123.456},"f4":"some string"}}
> {noformat}
> The key "f1" is missing in JSON string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes
[
https://issues.apache.org/jira/browse/SPARK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-23773:
---
Issue Type: Improvement (was: Bug)
> JacksonGenerator does not include keys that have null value for StructTypes
> ---
>
> Key: SPARK-23773
> URL: https://issues.apache.org/jira/browse/SPARK-23773
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.0.0, 2.3.0
>Reporter: Sergey
>Priority: Trivial
>
> When "toJSON" is called on a dataset, the result JSON string will not have
> keys displayed for StructTypes that have null value.
> Repro:
> {noformat}
> scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> ...
> scala> val df = sqlContext.sql(""" select NAMED_STRUCT('f1', null, 'f2',
> ARRAY(TRUE, FALSE), 'f3', MAP(123L, 123.456), 'f4', 'some string') as
> my_struct """)
> ...
> scala> df.toJSON.collect().foreach(println)
> {"my_struct":{"f2":[true,false],"f3":({"123":123.456},"f4":"some string"}}
> {noformat}
> The key "f1" is missing in JSON string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753005#comment-16753005 ] Sergey commented on SPARK-26688: Hi Imran, thanks for you reply. "Meanwhile devs start to apply this willy-nilly, as these configs tend to just keep getting built up over time " SLA could mitigate this problem. Every blacklisted node for a specific job slows it down in a long run. Anyway, dev would have to report / communicate with ops to resolve node issue. "Ideally, blacklisting and speculation should be able to prevent that problem" We are going to try out speculation but we are not there yet. > Provide configuration of initially blacklisted YARN nodes > - > > Key: SPARK-26688 > URL: https://issues.apache.org/jira/browse/SPARK-26688 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.0.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Introducing new config for initially blacklisted YARN nodes. > This came up in the apache spark user mailing list: > [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750498#comment-16750498 ] Sergey commented on SPARK-26688: Hi There! I'm very glad that the community paid attention to my question. Let me try to explain usecase There is 1K nodes cluster and jobs have performance degradation because of a single node. It's rather hard to convince Cluster Ops to decommission node because of "performance degradation". Imagine 10 dev teams chase single ops team for valid reason (node has problems) or because code has a bug or data is skewed or spots on the sun. We can't just decommission node because random dev complains. Simple solution: * rerun failed / delayed job and blacklist "problematic" node in advance. * Report about the problem if job works w/o anomalies. * ops collect complains about node and start to decommission it when "complains threshold" is reached. It's a rather low probability that many loosely coupled teams with loosely coupled jobs complain about a single node. Results * Ops are not spammed with a random requests from devs * devs are not blocked because of the really bad node. * it's very cheap for everyone to "blacklist" node during job submission w/o doing anything to node. > Provide configuration of initially blacklisted YARN nodes > - > > Key: SPARK-26688 > URL: https://issues.apache.org/jira/browse/SPARK-26688 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.0.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Introducing new config for initially blacklisted YARN nodes. > This came up in the apache spark user mailing list: > [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes
Sergey created SPARK-23773:
--
Summary: JacksonGenerator does not include keys that have null
value for StructTypes
Key: SPARK-23773
URL: https://issues.apache.org/jira/browse/SPARK-23773
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0, 2.0.0
Reporter: Sergey
When "toJSON" is called on a dataset, the result JSON string will not have keys
displayed for StructTypes that have null value.
Repro:
{noformat}
scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
...
scala> val df = sqlContext.sql(""" select NAMED_STRUCT('f1', null, 'f2',
ARRAY(TRUE, FALSE), 'f3', MAP(123L, 123.456), 'f4', 'some string') as my_struct
""")
...
scala> df.toJSON.collect().foreach(println)
{"my_struct":{"f2":[true,false],"f3":({"123":123.456},"f4":"some string"}}
{noformat}
The key "f1" is missing in JSON string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19900) [Standalone] Master registers application again when driver relaunched
[
https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959052#comment-15959052
]
Sergey commented on SPARK-19900:
This issue does not reproduced in spark 2.1.0 version.
> [Standalone] Master registers application again when driver relaunched
> --
>
> Key: SPARK-19900
> URL: https://issues.apache.org/jira/browse/SPARK-19900
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Spark Core
>Affects Versions: 1.6.2
> Environment: Centos 6.5, spark standalone
>Reporter: Sergey
>Priority: Critical
> Labels: Spark, network, standalone, supervise
>
> I've found some problems when node, where driver is running, has unstable
> network. A situation is possible when two identical applications are running
> on a cluster.
> *Steps to Reproduce:*
> # prepare 3 node. One for the spark master and two for the spark workers.
> # submit an application with parameter spark.driver.supervise = true
> # go to the node where driver is running (for example spark-worker-1) and
> close 7077 port
> {code}
> # iptables -A OUTPUT -p tcp --dport 7077 -j DROP
> {code}
> # wait more 60 seconds
> # look at the spark master UI
> There are two spark applications and one driver. The new application has
> WAITING state and the second application has RUNNING state. Driver has
> RUNNING or RELAUNCHING state (It depends on the resources available, as I
> understand it) and it launched on other node (for example spark-worker-2)
> # open the port
> {code}
> # iptables -D OUTPUT -p tcp --dport 7077 -j DROP
> {code}
> # look an the spark UI again
> There are no changes
> In addition, if you look at the processes on the node spark-worker-1
> {code}
> # ps ax | grep spark
> {code}
> you will see that the old driver is still working!
> *Spark master logs:*
> {code}
> 17/03/10 05:26:27 WARN Master: Removing
> worker-20170310052240-spark-worker-1-35039 because we got no heartbeat in 60
> seconds
> 17/03/10 05:26:27 INFO Master: Removing worker
> worker-20170310052240-spark-worker-1-35039 on spark-worker-1:35039
> 17/03/10 05:26:27 INFO Master: Telling app of lost executor: 1
> 17/03/10 05:26:27 INFO Master: Telling app of lost executor: 0
> 17/03/10 05:26:27 INFO Master: Re-launching driver-20170310052347-
> 17/03/10 05:26:27 INFO Master: Launching driver driver-20170310052347- on
> worker worker-20170310052411-spark-worker-2-40473
> 17/03/10 05:26:35 INFO Master: Registering app TestApplication
> 17/03/10 05:26:35 INFO Master: Registered app TestApplication with ID
> app-20170310052635-0001
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got status update for unknown executor
> app-20170310052354-/1
> 17/03/10 05:31:07 WARN Master: Got status update for unknown executor
> app-20170310052354-/0
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
>
[jira] [Updated] (SPARK-19900) [Standalone] Master registers application again when driver relaunched
[
https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-19900:
---
Priority: Critical (was: Major)
> [Standalone] Master registers application again when driver relaunched
> --
>
> Key: SPARK-19900
> URL: https://issues.apache.org/jira/browse/SPARK-19900
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Spark Core
>Affects Versions: 1.6.2
> Environment: Centos 6.5, spark standalone
>Reporter: Sergey
>Priority: Critical
> Labels: Spark, network, standalone, supervise
>
> I've found some problems when node, where driver is running, has unstable
> network. A situation is possible when two identical applications are running
> on a cluster.
> *Steps to Reproduce:*
> # prepare 3 node. One for the spark master and two for the spark workers.
> # submit an application with parameter spark.driver.supervise = true
> # go to the node where driver is running (for example spark-worker-1) and
> close 7077 port
> {code}
> # iptables -A OUTPUT -p tcp --dport 7077 -j DROP
> {code}
> # wait more 60 seconds
> # look at the spark master UI
> There are two spark applications and one driver. The new application has
> WAITING state and the second application has RUNNING state. Driver has
> RUNNING or RELAUNCHING state (It depends on the resources available, as I
> understand it) and it launched on other node (for example spark-worker-2)
> # open the port
> {code}
> # iptables -D OUTPUT -p tcp --dport 7077 -j DROP
> {code}
> # look an the spark UI again
> There are no changes
> In addition, if you look at the processes on the node spark-worker-1
> {code}
> # ps ax | grep spark
> {code}
> you will see that the old driver is still working!
> *Spark master logs:*
> {code}
> 17/03/10 05:26:27 WARN Master: Removing
> worker-20170310052240-spark-worker-1-35039 because we got no heartbeat in 60
> seconds
> 17/03/10 05:26:27 INFO Master: Removing worker
> worker-20170310052240-spark-worker-1-35039 on spark-worker-1:35039
> 17/03/10 05:26:27 INFO Master: Telling app of lost executor: 1
> 17/03/10 05:26:27 INFO Master: Telling app of lost executor: 0
> 17/03/10 05:26:27 INFO Master: Re-launching driver-20170310052347-
> 17/03/10 05:26:27 INFO Master: Launching driver driver-20170310052347- on
> worker worker-20170310052411-spark-worker-2-40473
> 17/03/10 05:26:35 INFO Master: Registering app TestApplication
> 17/03/10 05:26:35 INFO Master: Registered app TestApplication with ID
> app-20170310052635-0001
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got status update for unknown executor
> app-20170310052354-/1
> 17/03/10 05:31:07 WARN Master: Got status update for unknown executor
> app-20170310052354-/0
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
> worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
> 17/03/10 05:31:07 WARN
[jira] [Updated] (SPARK-19900) [Standalone] Master registers application again when driver relaunched
[
https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-19900:
---
Description:
I've found some problems when node, where driver is running, has unstable
network. A situation is possible when two identical applications are running on
a cluster.
*Steps to Reproduce:*
# prepare 3 node. One for the spark master and two for the spark workers.
# submit an application with parameter spark.driver.supervise = true
# go to the node where driver is running (for example spark-worker-1) and close
7077 port
{code}
# iptables -A OUTPUT -p tcp --dport 7077 -j DROP
{code}
# wait more 60 seconds
# look at the spark master UI
There are two spark applications and one driver. The new application has
WAITING state and the second application has RUNNING state. Driver has RUNNING
or RELAUNCHING state (It depends on the resources available, as I understand
it) and it launched on other node (for example spark-worker-2)
# open the port
{code}
# iptables -D OUTPUT -p tcp --dport 7077 -j DROP
{code}
# look an the spark UI again
There are no changes
In addition, if you look at the processes on the node spark-worker-1
{code}
# ps ax | grep spark
{code}
you will see that the old driver is still working!
*Spark master logs:*
{code}
17/03/10 05:26:27 WARN Master: Removing
worker-20170310052240-spark-worker-1-35039 because we got no heartbeat in 60
seconds
17/03/10 05:26:27 INFO Master: Removing worker
worker-20170310052240-spark-worker-1-35039 on spark-worker-1:35039
17/03/10 05:26:27 INFO Master: Telling app of lost executor: 1
17/03/10 05:26:27 INFO Master: Telling app of lost executor: 0
17/03/10 05:26:27 INFO Master: Re-launching driver-20170310052347-
17/03/10 05:26:27 INFO Master: Launching driver driver-20170310052347- on
worker worker-20170310052411-spark-worker-2-40473
17/03/10 05:26:35 INFO Master: Registering app TestApplication
17/03/10 05:26:35 INFO Master: Registered app TestApplication with ID
app-20170310052635-0001
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got status update for unknown executor
app-20170310052354-/1
17/03/10 05:31:07 WARN Master: Got status update for unknown executor
app-20170310052354-/0
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat
[jira] [Updated] (SPARK-19900) [Standalone] Master registers application again when driver relaunched
[
https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-19900:
---
Description:
I've found some problems when node, where driver is running, has unstable
network. A situation is possible when two identical applications are running on
a cluster.
*Steps to Reproduce:*
# prepare 3 node. One for the spark master and two for the spark workers.
# submit an application with parameter spark.driver.supervise = true
# go to the node where driver is running (for example spark-worker-1) and close
7077 port
{code}
# iptables -A OUTPUT -p tcp --dport 7077 -j DROP
{code}
# wait more 60 seconds
# look at the spark master UI
There are two spark applications and one driver. The new application has
WAITING state and the second application has RUNNING state. Driver has RUNNING
or RELAUNCHING state (It depends on the resources available, as I understand
it) and it launched on other node (for example spark-worker-2)
# open the port
{code}
# iptables -D OUTPUT -p tcp --dport 7077 -j DROP
{code}
# look an the spark UI again
There are no changes
In addition, if you look at the processes on the node spark-worker-1
{code}
# ps ax | grep spark
{code}
you will see that the old driver is still working!
*Spark master logs:*
{code}
17/03/10 05:26:27 WARN Master: Removing
worker-20170310052240-spark-worker-1-35039 because we got no heartbeat in 60
seconds
17/03/10 05:26:27 INFO Master: Removing worker
worker-20170310052240-spark-worker-1-35039 on spark-worker-1:35039
17/03/10 05:26:27 INFO Master: Telling app of lost executor: 1
17/03/10 05:26:27 INFO Master: Telling app of lost executor: 0
17/03/10 05:26:27 INFO Master: Re-launching driver-20170310052347-
17/03/10 05:26:27 INFO Master: Launching driver driver-20170310052347- on
worker worker-20170310052411-spark-worker-2-40473
17/03/10 05:26:35 INFO Master: Registering app TestApplication
17/03/10 05:26:35 INFO Master: Registered app TestApplication with ID
app-20170310052635-0001
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got status update for unknown executor
app-20170310052354-/1
17/03/10 05:31:07 WARN Master: Got status update for unknown executor
app-20170310052354-/0
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat from unregistered worker
worker-20170310052240-spark-worker-1-35039. Asking it to re-register.
17/03/10 05:31:07 WARN Master: Got heartbeat
[jira] [Updated] (SPARK-19900) [Standalone] Master registers application again when driver relaunched
[
https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey updated SPARK-19900:
---
Description:
I've found some problems when node, where driver is running, has unstable
network. A situation is possible when two identical applications are running on
a cluster.
*Steps to Reproduce:*
# prepare 3 node. One for the spark master and two for the spark workers.
# submit an application with parameter spark.driver.supervise = true
# go to the node where driver is running (for example spark-worker-1) and close
7077 port
{code}
# iptables -A OUTPUT -p tcp --dport 7077 -j DROP
{code}
# wait more 60 seconds
# look at the spark master UI
There are two spark applications and one driver. The new application has
WAITING state and the second application has RUNNING state. Driver has RUNNING
or RELAUNCHING state (It depends on the resources available, as I understand
it) and it launched on other node (for example spark-worker-2)
# open the port
{code}
# iptables -D OUTPUT -p tcp --dport 7077 -j DROP
{code}
# look an the spark UI again
There are no changes
In addition, if you look at the processes on the node spark-worker-1
{code}
# ps ax | grep spark
{code}
you will see that the old driver is still working!
was:
I've found some problems when node, where driver is running, has unstable
network. A situation is possible when two identical applications are running on
a cluster.
*Steps to Reproduce:*
# prepare 3 node. One for the spark master and two for the spark workers.
# submit an application with parameter spark.driver.supervise = true
# go to the node where driver is running (for example spark-worker-1) and close
7077 port
{code}
# iptables -A OUTPUT -p tcp --dport 7077 -j DROP
{code}
# wait more 60 seconds
# look at the spark master UI
There are two spark applications and one driver. The new application has
WAITING state and the second application has RUNNING state. Driver has RUNNING
or RELAUNCHING state (It depends on the resources available, as I understand
it) and it launched on other node (for example spark-worker-2)
# open the port
{code}
# iptables -D OUTPUT -p tcp --dport 7077 -j DROP
{code}
# look an the spark UI again
There are no changes
> [Standalone] Master registers application again when driver relaunched
> --
>
> Key: SPARK-19900
> URL: https://issues.apache.org/jira/browse/SPARK-19900
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Spark Core
>Affects Versions: 1.6.2
> Environment: Centos 6.5, spark standalone
>Reporter: Sergey
> Labels: Spark, network, standalone, supervise
>
> I've found some problems when node, where driver is running, has unstable
> network. A situation is possible when two identical applications are running
> on a cluster.
> *Steps to Reproduce:*
> # prepare 3 node. One for the spark master and two for the spark workers.
> # submit an application with parameter spark.driver.supervise = true
> # go to the node where driver is running (for example spark-worker-1) and
> close 7077 port
> {code}
> # iptables -A OUTPUT -p tcp --dport 7077 -j DROP
> {code}
> # wait more 60 seconds
> # look at the spark master UI
> There are two spark applications and one driver. The new application has
> WAITING state and the second application has RUNNING state. Driver has
> RUNNING or RELAUNCHING state (It depends on the resources available, as I
> understand it) and it launched on other node (for example spark-worker-2)
> # open the port
> {code}
> # iptables -D OUTPUT -p tcp --dport 7077 -j DROP
> {code}
> # look an the spark UI again
> There are no changes
> In addition, if you look at the processes on the node spark-worker-1
> {code}
> # ps ax | grep spark
> {code}
> you will see that the old driver is still working!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19900) [Standalone] Master registers application again when driver relaunched
Sergey created SPARK-19900:
--
Summary: [Standalone] Master registers application again when
driver relaunched
Key: SPARK-19900
URL: https://issues.apache.org/jira/browse/SPARK-19900
Project: Spark
Issue Type: Bug
Components: Deploy, Spark Core
Affects Versions: 1.6.2
Environment: Centos 6.5, spark standalone
Reporter: Sergey
I've found some problems when node, where driver is running, has unstable
network. A situation is possible when two identical applications are running on
a cluster.
*Steps to Reproduce:*
# prepare 3 node. One for the spark master and two for the spark workers.
# submit an application with parameter spark.driver.supervise = true
# go to the node where driver is running (for example spark-worker-1) and close
7077 port
{code}
# iptables -A OUTPUT -p tcp --dport 7077 -j DROP
{code}
# wait more 60 seconds
# look at the spark master UI
There are two spark applications and one driver. The new application has
WAITING state and the second application has RUNNING state. Driver has RUNNING
or RELAUNCHING state (It depends on the resources available, as I understand
it) and it launched on other node (for example spark-worker-2)
# open the port
{code}
# iptables -D OUTPUT -p tcp --dport 7077 -j DROP
{code}
# look an the spark UI again
There are no changes
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14663) Parse escape sequences in spark-defaults.conf
[ https://issues.apache.org/jira/browse/SPARK-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-14663: --- Priority: Minor (was: Major) > Parse escape sequences in spark-defaults.conf > - > > Key: SPARK-14663 > URL: https://issues.apache.org/jira/browse/SPARK-14663 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.1 >Reporter: Sergey >Priority: Minor > > I am trying to specify > spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, > to set it to "\n" (the #10 character). I know how to do it in > sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep > using sc.textFile (because it also works with zipped files). > However, I can't find a way to accomplish it. > I have tried > spark.hadoop.textinputformat.record.delimiter \n > spark.hadoop.textinputformat.record.delimiter '\n' > spark.hadoop.textinputformat.record.delimiter "\n" > spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and > the letter n) > spark.hadoop.textinputformat.record.delimiter > (just pressing enter) > None of them works. I check in sc._conf.getAll(), and none of them gives me > the right result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14663) Parse escape sequences in spark-defaults.conf
[ https://issues.apache.org/jira/browse/SPARK-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243244#comment-15243244 ] Sergey commented on SPARK-14663: UPDATE: I have found a different solution for my particular issue (new APIHadoopFile happens to work with zip files even if passed "org.apache.hadoop.mapreduce.lib.input.TextInputFormat", not "com.cotdp.hadoop.ZipFileInputFormat"). However, the issue remains that a value in spark-defaults.conf cannot be set to "\n". > Parse escape sequences in spark-defaults.conf > - > > Key: SPARK-14663 > URL: https://issues.apache.org/jira/browse/SPARK-14663 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.1 >Reporter: Sergey > > I am trying to specify > spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, > to set it to "\n" (the #10 character). I know how to do it in > sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep > using sc.textFile (because it also works with zipped files). > However, I can't find a way to accomplish it. > I have tried > spark.hadoop.textinputformat.record.delimiter \n > spark.hadoop.textinputformat.record.delimiter '\n' > spark.hadoop.textinputformat.record.delimiter "\n" > spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and > the letter n) > spark.hadoop.textinputformat.record.delimiter > (just pressing enter) > None of them works. I check in sc._conf.getAll(), and none of them gives me > the right result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14663) Parse escape sequences in spark-defaults.conf
[ https://issues.apache.org/jira/browse/SPARK-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-14663: --- Description: I am trying to specify spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, to set it to "\n" (the #10 character). I know how to do it in sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep using sc.textFile (because it also works with zipped files). However, I can't find a way to accomplish it. I have tried spark.hadoop.textinputformat.record.delimiter \n spark.hadoop.textinputformat.record.delimiter '\n' spark.hadoop.textinputformat.record.delimiter "\n" spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and the letter n) spark.hadoop.textinputformat.record.delimiter (just pressing enter) None of them works. I check in sc._conf.getAll(), and none of them gives me the right result. was: I am trying to specify spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, to set it to "\n" (the #10 character). I know how to do it in sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep using sc.textFile (because it also works with zipped files). However, I can't find a way to accomplish it. I have tried spark.hadoop.textinputformat.record.delimiter \n spark.hadoop.textinputformat.record.delimiter '\n' spark.hadoop.textinputformat.record.delimiter "\n" spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and the letter n) spark.hadoop.textinputformat.record.delimiter (just pressing enter) None of them works. I check in sc._conf.getAll(), and none of them gives me the right result. > Parse escape sequences in spark-defaults.conf > - > > Key: SPARK-14663 > URL: https://issues.apache.org/jira/browse/SPARK-14663 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.1 >Reporter: Sergey > > I am trying to specify > spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, > to set it to "\n" (the #10 character). I know how to do it in > sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep > using sc.textFile (because it also works with zipped files). > However, I can't find a way to accomplish it. > I have tried > spark.hadoop.textinputformat.record.delimiter \n > spark.hadoop.textinputformat.record.delimiter '\n' > spark.hadoop.textinputformat.record.delimiter "\n" > spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and > the letter n) > spark.hadoop.textinputformat.record.delimiter > (just pressing enter) > None of them works. I check in sc._conf.getAll(), and none of them gives me > the right result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14663) Parse escape sequences in spark-defaults.conf
[ https://issues.apache.org/jira/browse/SPARK-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-14663: --- Description: I am trying to specify spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, to set it to "\n" (the #10 character). I know how to do it in sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep using sc.textFile (because it also works with zipped files). However, I can't find a way to accomplish it. I have tried spark.hadoop.textinputformat.record.delimiter \n spark.hadoop.textinputformat.record.delimiter '\n' spark.hadoop.textinputformat.record.delimiter "\n" spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and the letter n) spark.hadoop.textinputformat.record.delimiter (just pressing enter) None of them works. I check in sc._conf.getAll(), and none of them gives me the right result. was: I am trying to specify spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, to set it to "\n" (the #10 character). I know how to do it in sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep using sc.textFile (because it also works with zipped files). However, I can't find a way to accomplish it. I have tried spark.hadoop.textinputformat.record.delimiter \n spark.hadoop.textinputformat.record.delimiter '\n' spark.hadoop.textinputformat.record.delimiter "\n" spark.hadoop.textinputformat.record.delimiter \\n spark.hadoop.textinputformat.record.delimiter (just pressing enter) None of them works. I check in sc._conf.getAll(), and none of them gives me the right result. > Parse escape sequences in spark-defaults.conf > - > > Key: SPARK-14663 > URL: https://issues.apache.org/jira/browse/SPARK-14663 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.1 >Reporter: Sergey > > I am trying to specify > spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, > to set it to "\n" (the #10 character). I know how to do it in > sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep > using sc.textFile (because it also works with zipped files). > However, I can't find a way to accomplish it. > I have tried > spark.hadoop.textinputformat.record.delimiter \n > spark.hadoop.textinputformat.record.delimiter '\n' > spark.hadoop.textinputformat.record.delimiter "\n" > spark.hadoop.textinputformat.record.delimiter \\n (that's two slashes and > the letter n) > spark.hadoop.textinputformat.record.delimiter (just pressing enter) > None of them works. I check in sc._conf.getAll(), and none of them gives me > the right result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-14663) Parse escape sequences in spark-defaults.conf
Sergey created SPARK-14663: -- Summary: Parse escape sequences in spark-defaults.conf Key: SPARK-14663 URL: https://issues.apache.org/jira/browse/SPARK-14663 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.6.1 Reporter: Sergey I am trying to specify spark.hadoop.textinputformat.record.delimiter in spark-defaults.conf, namely, to set it to "\n" (the #10 character). I know how to do it in sc.newAPIHadoopFile, but I'd like to set it in configuration, so I can keep using sc.textFile (because it also works with zipped files). However, I can't find a way to accomplish it. I have tried spark.hadoop.textinputformat.record.delimiter \n spark.hadoop.textinputformat.record.delimiter '\n' spark.hadoop.textinputformat.record.delimiter "\n" spark.hadoop.textinputformat.record.delimiter \\n spark.hadoop.textinputformat.record.delimiter (just pressing enter) None of them works. I check in sc._conf.getAll(), and none of them gives me the right result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-2388) Streaming from multiple different Kafka topics is problematic
Sergey created SPARK-2388: - Summary: Streaming from multiple different Kafka topics is problematic Key: SPARK-2388 URL: https://issues.apache.org/jira/browse/SPARK-2388 Project: Spark Issue Type: Improvement Components: Streaming Affects Versions: 1.0.0 Reporter: Sergey Fix For: 1.0.1 Default way of creating stream out of Kafka source would be as val stream = KafkaUtils.createStream(ssc,localhost:2181,logs, Map(retarget - 2,datapair - 2)) However, if two topics - in this case retarget and datapair - are very different, there is no way to set up different filter, mapping functions, etc), as they are effectively merged. However, instance of KafkaInputDStream, created with this call internally calls ConsumerConnector.createMessageStream() which returns *map* of KafkaStreams, keyed by topic. It would be great if this map would be exposed somehow, so aforementioned call val streamS = KafkaUtils.createStreamS(...) returned map of streams. Regards, Sergey Malov Collective Media -- This message was sent by Atlassian JIRA (v6.2#6252)
