[jira] [Updated] (SPARK-20464) Add a job group and an informative description for streaming queries
[ https://issues.apache.org/jira/browse/SPARK-20464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-20464: - Summary: Add a job group and an informative description for streaming queries (was: Add a job group and an informative job description for streaming queries) > Add a job group and an informative description for streaming queries > > > Key: SPARK-20464 > URL: https://issues.apache.org/jira/browse/SPARK-20464 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.2.0 >Reporter: Kunal Khamar > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20464) Add a job group and an informative job description for streaming queries
Kunal Khamar created SPARK-20464: Summary: Add a job group and an informative job description for streaming queries Key: SPARK-20464 URL: https://issues.apache.org/jira/browse/SPARK-20464 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.2.0 Reporter: Kunal Khamar -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20164) AnalysisException not tolerant of null query plan
[ https://issues.apache.org/jira/browse/SPARK-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-20164: - Description: The query plan in an AnalysisException may be null when an AnalysisException object is serialized and then deserialized, since plan is marked @transient. Or when someone throws an AnalysisException with a null query plan (which should not happen). def getMessage is not tolerant of this and throws a NullPointerException, leading to loss of information about the original exception. The fix is to add a null check in getMessage. was: The query plan in an `AnalysisException` may be `null` when an `AnalysisException` object is serialized and then deserialized, since `plan` is marked `@transient`. Or when someone throws an `AnalysisException` with a null query plan (which should not happen). `def getMessage` is not tolerant of this and throws a `NullPointerException`, leading to loss of information about the original exception. The fix is to add a `null` check in `getMessage`. > AnalysisException not tolerant of null query plan > - > > Key: SPARK-20164 > URL: https://issues.apache.org/jira/browse/SPARK-20164 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kunal Khamar >Assignee: Kunal Khamar > Fix For: 2.2.0, 2.1.2 > > > The query plan in an AnalysisException may be null when an AnalysisException > object is serialized and then deserialized, since plan is marked @transient. > Or when someone throws an AnalysisException with a null query plan (which > should not happen). > def getMessage is not tolerant of this and throws a NullPointerException, > leading to loss of information about the original exception. > The fix is to add a null check in getMessage. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20164) AnalysisException not tolerant of null query plan
[ https://issues.apache.org/jira/browse/SPARK-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-20164: - Description: The query plan in an `AnalysisException` may be `null` when an `AnalysisException` object is serialized and then deserialized, since `plan` is marked `@transient`. Or when someone throws an `AnalysisException` with a null query plan (which should not happen). `def getMessage` is not tolerant of this and throws a `NullPointerException`, leading to loss of information about the original exception. The fix is to add a `null` check in `getMessage`. was:When someone throws an AnalysisException with a null query plan (which ideally no one should), getMessage is not tolerant of this and throws a null pointer exception, leading to loss of information about original exception. > AnalysisException not tolerant of null query plan > - > > Key: SPARK-20164 > URL: https://issues.apache.org/jira/browse/SPARK-20164 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kunal Khamar >Assignee: Kunal Khamar > Fix For: 2.2.0, 2.1.2 > > > The query plan in an `AnalysisException` may be `null` when an > `AnalysisException` object is serialized and then deserialized, since `plan` > is marked `@transient`. Or when someone throws an `AnalysisException` with a > null query plan (which should not happen). > `def getMessage` is not tolerant of this and throws a `NullPointerException`, > leading to loss of information about the original exception. > The fix is to add a `null` check in `getMessage`. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20164) AnalysisException not tolerant of null query plan
Kunal Khamar created SPARK-20164: Summary: AnalysisException not tolerant of null query plan Key: SPARK-20164 URL: https://issues.apache.org/jira/browse/SPARK-20164 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.0 Reporter: Kunal Khamar When someone throws an AnalysisException with a null query plan (which ideally no one should), getMessage is not tolerant of this and throws a null pointer exception, leading to loss of information about original exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20147) Cloning SessionState does not clone streaming query listeners
[ https://issues.apache.org/jira/browse/SPARK-20147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-20147: - Description: Cloning session should clone StreamingQueryListeners registered on the StreamingQueryListenerBus. Similar to SPARK-20048, https://github.com/apache/spark/pull/17379 was:Cloning session should clone StreamingQueryListeners registered on the StreamingQueryListenerBus. > Cloning SessionState does not clone streaming query listeners > - > > Key: SPARK-20147 > URL: https://issues.apache.org/jira/browse/SPARK-20147 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Kunal Khamar > > Cloning session should clone StreamingQueryListeners registered on the > StreamingQueryListenerBus. > Similar to SPARK-20048, https://github.com/apache/spark/pull/17379 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20147) Cloning SessionState does not clone streaming query listeners
[ https://issues.apache.org/jira/browse/SPARK-20147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-20147: - Description: Cloning session should clone StreamingQueryListeners registered on the StreamingQueryListenerBus. > Cloning SessionState does not clone streaming query listeners > - > > Key: SPARK-20147 > URL: https://issues.apache.org/jira/browse/SPARK-20147 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Kunal Khamar > > Cloning session should clone StreamingQueryListeners registered on the > StreamingQueryListenerBus. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20147) Cloning SessionState does not clone streaming query listeners
Kunal Khamar created SPARK-20147: Summary: Cloning SessionState does not clone streaming query listeners Key: SPARK-20147 URL: https://issues.apache.org/jira/browse/SPARK-20147 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.1.0 Reporter: Kunal Khamar -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20051) Fix StreamSuite.recover from v2.1 checkpoint failing with IOException
[ https://issues.apache.org/jira/browse/SPARK-20051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-20051: - Description: There is a race condition between calling stop on a streaming query and deleting directories in withTempDir that causes test to fail, fixing to do lazy deletion using delete on shutdown JVM hook. (was: There is a race condition with deleting directories in withTempDir that causes test to fail, fixing to do lazy deletion using delete on shutdown JVM hook.) > Fix StreamSuite.recover from v2.1 checkpoint failing with IOException > - > > Key: SPARK-20051 > URL: https://issues.apache.org/jira/browse/SPARK-20051 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.2.0 >Reporter: Kunal Khamar > > There is a race condition between calling stop on a streaming query and > deleting directories in withTempDir that causes test to fail, fixing to do > lazy deletion using delete on shutdown JVM hook. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20051) Fix StreamSuite.recover from v2.1 checkpoint failing with IOException
Kunal Khamar created SPARK-20051: Summary: Fix StreamSuite.recover from v2.1 checkpoint failing with IOException Key: SPARK-20051 URL: https://issues.apache.org/jira/browse/SPARK-20051 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.2.0 Reporter: Kunal Khamar There is a race condition with deleting directories in withTempDir that causes test to fail, fixing to do lazy deletion using delete on shutdown JVM hook. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20048) Cloning SessionState does not clone query execution listeners
Kunal Khamar created SPARK-20048: Summary: Cloning SessionState does not clone query execution listeners Key: SPARK-20048 URL: https://issues.apache.org/jira/browse/SPARK-20048 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.0 Reporter: Kunal Khamar -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19540) Add ability to clone SparkSession with an identical copy of the SessionState
[ https://issues.apache.org/jira/browse/SPARK-19540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19540: - Summary: Add ability to clone SparkSession with an identical copy of the SessionState (was: Add ability to clone SparkSession wherein cloned session has an identical copy of the SessionState) > Add ability to clone SparkSession with an identical copy of the SessionState > > > Key: SPARK-19540 > URL: https://issues.apache.org/jira/browse/SPARK-19540 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kunal Khamar > > Forking a newSession() from SparkSession currently makes a new SparkSession > that does not retain SessionState (i.e. temporary tables, SQL config, > registered functions etc.) This change adds a method cloneSession() which > creates a new SparkSession with a copy of the parent's SessionState. > Subsequent changes to base session are not propagated to cloned session, > clone is independent after creation. > If the base is changed after clone has been created, say user registers new > UDF, then the new UDF will not be available inside the clone. Same goes for > configs and temp tables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19540) Add ability to clone SparkSession wherein cloned session has an identical copy of the SessionState
[ https://issues.apache.org/jira/browse/SPARK-19540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19540: - Summary: Add ability to clone SparkSession wherein cloned session has an identical copy of the SessionState (was: Add ability to clone SparkSession wherein cloned session has a reference to SharedState and an identical copy of the SessionState) > Add ability to clone SparkSession wherein cloned session has an identical > copy of the SessionState > -- > > Key: SPARK-19540 > URL: https://issues.apache.org/jira/browse/SPARK-19540 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kunal Khamar > > Forking a newSession() from SparkSession currently makes a new SparkSession > that does not retain SessionState (i.e. temporary tables, SQL config, > registered functions etc.) This change adds a method cloneSession() which > creates a new SparkSession with a copy of the parent's SessionState. > Subsequent changes to base session are not propagated to cloned session, > clone is independent after creation. > If the base is changed after clone has been created, say user registers new > UDF, then the new UDF will not be available inside the clone. Same goes for > configs and temp tables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19873) If the user changes the number of shuffle partitions between batches, Streaming aggregation will fail.
[ https://issues.apache.org/jira/browse/SPARK-19873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19873: - Summary: If the user changes the number of shuffle partitions between batches, Streaming aggregation will fail. (was: If the user changes the shuffle partition number between batches, Streaming aggregation will fail.) > If the user changes the number of shuffle partitions between batches, > Streaming aggregation will fail. > -- > > Key: SPARK-19873 > URL: https://issues.apache.org/jira/browse/SPARK-19873 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Kunal Khamar > > If the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > Here are some possible cases: > - Change "spark.sql.shuffle.partitions" > - Use "repartition" and change the partition number in codes > - RangePartitioner doesn't generate deterministic partitions. Right now it's > safe as we disallow sort before aggregation. Not sure if we will add some > operators using RangePartitioner in future. > Fix: > Record # shuffle partitions in offset log and enforce in next batch -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19873) If the user changes the shuffle partition number between batches, Streaming aggregation will fail.
[ https://issues.apache.org/jira/browse/SPARK-19873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19873: - Description: If the user changes the shuffle partition number between batches, Streaming aggregation will fail. Here are some possible cases: - Change "spark.sql.shuffle.partitions" - Use "repartition" and change the partition number in codes - RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future. Fix: Record # shuffle partition in offset log and enforce in next batch was: It the user changes the shuffle partition number between batches, Streaming aggregation will fail. Here are some possible cases: - Change "spark.sql.shuffle.partitions" - Use "repartition" and change the partition number in codes - RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future. Fix: Record # shuffle partition in offset log and enforce in next batch > If the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > -- > > Key: SPARK-19873 > URL: https://issues.apache.org/jira/browse/SPARK-19873 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Kunal Khamar > > If the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > Here are some possible cases: > - Change "spark.sql.shuffle.partitions" > - Use "repartition" and change the partition number in codes > - RangePartitioner doesn't generate deterministic partitions. Right now it's > safe as we disallow sort before aggregation. Not sure if we will add some > operators using RangePartitioner in future. > Fix: > Record # shuffle partition in offset log and enforce in next batch -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19873) If the user changes the shuffle partition number between batches, Streaming aggregation will fail.
[ https://issues.apache.org/jira/browse/SPARK-19873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19873: - Description: If the user changes the shuffle partition number between batches, Streaming aggregation will fail. Here are some possible cases: - Change "spark.sql.shuffle.partitions" - Use "repartition" and change the partition number in codes - RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future. Fix: Record # shuffle partitions in offset log and enforce in next batch was: If the user changes the shuffle partition number between batches, Streaming aggregation will fail. Here are some possible cases: - Change "spark.sql.shuffle.partitions" - Use "repartition" and change the partition number in codes - RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future. Fix: Record # shuffle partition in offset log and enforce in next batch > If the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > -- > > Key: SPARK-19873 > URL: https://issues.apache.org/jira/browse/SPARK-19873 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Kunal Khamar > > If the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > Here are some possible cases: > - Change "spark.sql.shuffle.partitions" > - Use "repartition" and change the partition number in codes > - RangePartitioner doesn't generate deterministic partitions. Right now it's > safe as we disallow sort before aggregation. Not sure if we will add some > operators using RangePartitioner in future. > Fix: > Record # shuffle partitions in offset log and enforce in next batch -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19873) If the user changes the shuffle partition number between batches, Streaming aggregation will fail.
[ https://issues.apache.org/jira/browse/SPARK-19873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19873: - Description: It the user changes the shuffle partition number between batches, Streaming aggregation will fail. Here are some possible cases: - Change "spark.sql.shuffle.partitions" - Use "repartition" and change the partition number in codes - RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future. Fix: Record # shuffle partition in offset log and enforce in next batch > If the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > -- > > Key: SPARK-19873 > URL: https://issues.apache.org/jira/browse/SPARK-19873 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Kunal Khamar > > It the user changes the shuffle partition number between batches, Streaming > aggregation will fail. > Here are some possible cases: > - Change "spark.sql.shuffle.partitions" > - Use "repartition" and change the partition number in codes > - RangePartitioner doesn't generate deterministic partitions. Right now it's > safe as we disallow sort before aggregation. Not sure if we will add some > operators using RangePartitioner in future. > Fix: > Record # shuffle partition in offset log and enforce in next batch -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19873) If the user changes the shuffle partition number between batches, Streaming aggregation will fail.
Kunal Khamar created SPARK-19873: Summary: If the user changes the shuffle partition number between batches, Streaming aggregation will fail. Key: SPARK-19873 URL: https://issues.apache.org/jira/browse/SPARK-19873 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.1.0 Reporter: Kunal Khamar -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19540) Add ability to clone SparkSession wherein cloned session has a reference to SharedState and an identical copy of the SessionState
[ https://issues.apache.org/jira/browse/SPARK-19540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19540: - Affects Version/s: (was: 2.1.1) 2.2.0 > Add ability to clone SparkSession wherein cloned session has a reference to > SharedState and an identical copy of the SessionState > - > > Key: SPARK-19540 > URL: https://issues.apache.org/jira/browse/SPARK-19540 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kunal Khamar > > Forking a newSession() from SparkSession currently makes a new SparkSession > that does not retain SessionState (i.e. temporary tables, SQL config, > registered functions etc.) This change adds a method cloneSession() which > creates a new SparkSession with a copy of the parent's SessionState. > Subsequent changes to base session are not propagated to cloned session, > clone is independent after creation. > If the base is changed after clone has been created, say user registers new > UDF, then the new UDF will not be available inside the clone. Same goes for > configs and temp tables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19540) Add ability to clone SparkSession wherein cloned session has a reference to SharedState and an identical copy of the SessionState
[ https://issues.apache.org/jira/browse/SPARK-19540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khamar updated SPARK-19540: - Issue Type: Improvement (was: Story) > Add ability to clone SparkSession wherein cloned session has a reference to > SharedState and an identical copy of the SessionState > - > > Key: SPARK-19540 > URL: https://issues.apache.org/jira/browse/SPARK-19540 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.1 >Reporter: Kunal Khamar > > Forking a newSession() from SparkSession currently makes a new SparkSession > that does not retain SessionState (i.e. temporary tables, SQL config, > registered functions etc.) This change adds a method cloneSession() which > creates a new SparkSession with a copy of the parent's SessionState. > Subsequent changes to base session are not propagated to cloned session, > clone is independent after creation. > If the base is changed after clone has been created, say user registers new > UDF, then the new UDF will not be available inside the clone. Same goes for > configs and temp tables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19540) Add ability to clone SparkSession wherein cloned session has a reference to SharedState and an identical copy of the SessionState
Kunal Khamar created SPARK-19540: Summary: Add ability to clone SparkSession wherein cloned session has a reference to SharedState and an identical copy of the SessionState Key: SPARK-19540 URL: https://issues.apache.org/jira/browse/SPARK-19540 Project: Spark Issue Type: Story Components: SQL Affects Versions: 2.1.1 Reporter: Kunal Khamar Forking a newSession() from SparkSession currently makes a new SparkSession that does not retain SessionState (i.e. temporary tables, SQL config, registered functions etc.) This change adds a method cloneSession() which creates a new SparkSession with a copy of the parent's SessionState. Subsequent changes to base session are not propagated to cloned session, clone is independent after creation. If the base is changed after clone has been created, say user registers new UDF, then the new UDF will not be available inside the clone. Same goes for configs and temp tables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
