David Ross created SPARK-6757:
---------------------------------
Summary: spark.sql.shuffle.partitions is global, not per connection
Key: SPARK-6757
URL: https://issues.apache.org/jira/browse/SPARK-6757
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.3.0
Reporter: David Ross
We are trying to use the {{spark.sql.shuffle.partitions}} parameter to handle
large queries differently from smaller queries. We expected that this parameter
would be respected per connection, but it seems to be global.
For example, in try this in two separate JDBC connections:
Connection 1:
{code}
SET spark.sql.shuffle.partitions=10;
SELECT * FROM some_table;
{code}
The correct number {{10}} was used.
Connection 2:
{code}
SET spark.sql.shuffle.partitions=100;
SELECT * FROM some_table;
{code}
The correct number {{100}} was used.
Back to connection 1:
{code}
SELECT * FROM some_table;
{code}
We expected the number {{10}} to be used but {{100}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
