[
https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-6962:
-----------------------------
Priority: Major (was: Blocker)
Until this is more vetted, reducing from Blocker.
It's hard to get any further without more info, like what was stuck where. Do
you have a stack dump? Or any code to reproduce?
This may be a duplicate of https://issues.apache.org/jira/browse/SPARK-4395 or
https://issues.apache.org/jira/browse/SPARK-5060 for example; it's worth
searching JIRA first.
> Spark gets stuck on a step, hangs forever - jobs do not complete
> ----------------------------------------------------------------
>
> Key: SPARK-6962
> URL: https://issues.apache.org/jira/browse/SPARK-6962
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0, 1.2.1, 1.3.0
> Reporter: Jon Chase
>
> Spark SQL queries (though this seems to be a Spark Core issue - I'm just
> using queries in the REPL to surface this, so I mention Spark SQL) hang
> indefinitely under certain (not totally understood) circumstances.
> This is resolved by setting spark.shuffle.blockTransferService=nio, which
> seems to point to netty as the issue. Netty was set as the default for the
> block transport layer in 1.2.0, which is when this issue started. Setting
> the service to nio allows queries to complete normally.
> I do not see this problem when running queries over smaller (~20 5MB files)
> datasets. When I increase the scope to include more data (several hundred
> ~5MB files), the queries will get through several steps but eventuall hang
> indefinitely.
> Here's the email chain regarding this issue, including stack traces:
> http://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/<cae61spfqt2y7d5vqzomzz2dmr-jx2c2zggcyky40npkjjx4...@mail.gmail.com>
> For context, here's the announcement regarding the block transfer service
> change:
> http://mail-archives.apache.org/mod_mbox/spark-dev/201411.mbox/<cabpqxssl04q+rbltp-d8w+z3atn+g-um6gmdgdnh-hzcvd-...@mail.gmail.com>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
