GitHub user rajeshbalamohan opened a pull request:
https://github.com/apache/spark/pull/12514
[SPARK-14521][SQL] StackOverflowError in Kryo when executing TPC-DS Qâ¦
## What changes were proposed in this pull request?
Observed stackOverflowError in Kryo when executing TPC-DS Query27. Spark
thrift server disables kryo reference tracking (if not specified in conf). When
"spark.kryo.referenceTracking" is set to true explicitly in
spark-defaults.conf, query executes successfully. Recent changes HashedRelation
could have introduced loops which would need
"spark.kryo.referenceTracking=true" in spark-thrift server. This PR addresses
this by setting referenceTracking to true in SparkSQLEnv
## How was this patch tested?
Manually running TPC-DS queries at 200 GB scale in multi node cluster.
Also ran
org.apache.spark.sql.hive.execution.HiveCompatibilitySuite,org.apache.spark.sql.hive.execution.HiveQuerySuite,org.apache.spark.sql.hive.execution.PruningSuite,org.apache.spark.sql.hive/CachedTableSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rajeshbalamohan/spark SPARK-14521
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12514.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12514
----
commit 59875f424aaf60aa90ca5a1006df8b5e20d4f83a
Author: Rajesh Balamohan <[email protected]>
Date: 2016-04-20T00:59:32Z
[SPARK-14521][SQL] StackOverflowError in Kryo when executing TPC-DS Query27
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
