[jira] [Assigned] (SPARK-44508) Add user guide for Python UDTFs
[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44508: Assignee: Allison Wang > Add user guide for Python UDTFs > --- > > Key: SPARK-44508 > URL: https://issues.apache.org/jira/browse/SPARK-44508 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 3.5.0 >Reporter: Allison Wang >Assignee: Allison Wang >Priority: Major > > Add documentation for Python UDTFs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44508) Add user guide for Python UDTFs
[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44508. -- Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull request 42272 [https://github.com/apache/spark/pull/42272] > Add user guide for Python UDTFs > --- > > Key: SPARK-44508 > URL: https://issues.apache.org/jira/browse/SPARK-44508 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 3.5.0 >Reporter: Allison Wang >Assignee: Allison Wang >Priority: Major > Fix For: 3.5.1, 4.0.0 > > > Add documentation for Python UDTFs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-43252) Assign a name to the error class _LEGACY_ERROR_TEMP_2016
[
https://issues.apache.org/jira/browse/SPARK-43252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762945#comment-17762945
]
Snoot.io commented on SPARK-43252:
--
User 'dengziming' has created a pull request for this issue:
https://github.com/apache/spark/pull/42846
> Assign a name to the error class _LEGACY_ERROR_TEMP_2016
>
>
> Key: SPARK-43252
> URL: https://issues.apache.org/jira/browse/SPARK-43252
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2016* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45104) Upgrade graphlib-dot.min.js from 0.6.4 to 1.0.2
[ https://issues.apache.org/jira/browse/SPARK-45104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762938#comment-17762938 ] Snoot.io commented on SPARK-45104: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/42853 > Upgrade graphlib-dot.min.js from 0.6.4 to 1.0.2 > --- > > Key: SPARK-45104 > URL: https://issues.apache.org/jira/browse/SPARK-45104 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Kent Yao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45104) Upgrade graphlib-dot.min.js from 0.6.4 to 1.0.2
Kent Yao created SPARK-45104: Summary: Upgrade graphlib-dot.min.js from 0.6.4 to 1.0.2 Key: SPARK-45104 URL: https://issues.apache.org/jira/browse/SPARK-45104 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 4.0.0 Reporter: Kent Yao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true
[
https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762935#comment-17762935
]
Snoot.io commented on SPARK-44805:
--
User 'bersprockets' has created a pull request for this issue:
https://github.com/apache/spark/pull/42850
> Data lost after union using
> spark.sql.parquet.enableNestedColumnVectorizedReader=true
> -
>
> Key: SPARK-44805
> URL: https://issues.apache.org/jira/browse/SPARK-44805
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.1, 3.4.1
> Environment: pySpark, linux, hadoop, parquet.
>Reporter: Jakub Wozniak
>Priority: Major
> Labels: correctness
>
> When union-ing two DataFrames read from parquet containing nested structures
> (2 fields of array types where one is double and second is integer) data from
> the second field seems to be lost (zeros are set instead).
> This seems to be the case only if nested vectorised reader is used
> (spark.sql.parquet.enableNestedColumnVectorizedReader=true).
> The following Python code reproduces the problem:
> {code:java}
> from pyspark.sql import SparkSession
> from pyspark.sql.types import *
> # PREPARING DATA
> data1 = []
> data2 = []
> for i in range(2):
> data1.append( (([1,2,3],[1,1,2]),i))
> data2.append( (([1.0,2.0,3.0],[1,1]),i+10))
> schema1 = StructType([
> StructField('value', StructType([
> StructField('f1', ArrayType(IntegerType()), True),
> StructField('f2', ArrayType(IntegerType()), True)
> ])),
> StructField('id', IntegerType(), True)
> ])
> schema2 = StructType([
> StructField('value', StructType([
> StructField('f1', ArrayType(DoubleType()), True),
> StructField('f2', ArrayType(IntegerType()), True)
> ])),
> StructField('id', IntegerType(), True)
> ])
> spark = SparkSession.builder.getOrCreate()
> data_dir = "/user//"
> df1 = spark.createDataFrame(data1, schema1)
> df1.write.mode('overwrite').parquet(data_dir + "data1")
> df2 = spark.createDataFrame(data2, schema2)
> df2.write.mode('overwrite').parquet(data_dir + "data2")
> # READING DATA
> parquet1 = spark.read.parquet(data_dir + "data1")
> parquet2 = spark.read.parquet(data_dir + "data2")
> # UNION
> out = parquet1.union(parquet2)
> parquet1.select("value.f2").distinct().show()
> out.select("value.f2").distinct().show()
> print(parquet1.collect())
> print(out.collect()) {code}
> Output:
> {code:java}
> +-+
> | f2|
> +-+
> |[1, 1, 2]|
> +-+
> +-+
> | f2|
> +-+
> |[0, 0, 0]|
> | [1, 1]|
> +-+
> [
> Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=0),
> Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=1)
> ]
> [
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=0),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=1),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=10),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=11)
> ] {code}
> Please notice that values for the field f2 are lost after the union is done.
> This only happens when this data is read from parquet files.
> Could you please look into this?
> Best regards,
> Jakub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45103) Update ORC to 1.8.5
[ https://issues.apache.org/jira/browse/SPARK-45103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45103: - Assignee: Dongjoon Hyun > Update ORC to 1.8.5 > --- > > Key: SPARK-45103 > URL: https://issues.apache.org/jira/browse/SPARK-45103 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.4.2 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45103) Update ORC to 1.8.5
[ https://issues.apache.org/jira/browse/SPARK-45103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45103. --- Fix Version/s: 3.4.2 Resolution: Fixed Issue resolved by pull request 42851 [https://github.com/apache/spark/pull/42851] > Update ORC to 1.8.5 > --- > > Key: SPARK-45103 > URL: https://issues.apache.org/jira/browse/SPARK-45103 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.4.2 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.4.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44732) Port the initial implementation of Spark XML data source
[ https://issues.apache.org/jira/browse/SPARK-44732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762891#comment-17762891 ] Ignite TC Bot commented on SPARK-44732: --- User 'srowen' has created a pull request for this issue: https://github.com/apache/spark/pull/42844 > Port the initial implementation of Spark XML data source > > > Key: SPARK-44732 > URL: https://issues.apache.org/jira/browse/SPARK-44732 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45103) Update ORC to 1.8.5
[ https://issues.apache.org/jira/browse/SPARK-45103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45103: -- Summary: Update ORC to 1.8.5 (was: Update ORC to 1.8.4) > Update ORC to 1.8.5 > --- > > Key: SPARK-45103 > URL: https://issues.apache.org/jira/browse/SPARK-45103 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.4.2 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45103) Update ORC to 1.8.4
Dongjoon Hyun created SPARK-45103: - Summary: Update ORC to 1.8.4 Key: SPARK-45103 URL: https://issues.apache.org/jira/browse/SPARK-45103 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.4.2 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45084) ProgressReport should include an accurate effective shuffle partition number
[ https://issues.apache.org/jira/browse/SPARK-45084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762825#comment-17762825 ] Hudson commented on SPARK-45084: User 'siying' has created a pull request for this issue: https://github.com/apache/spark/pull/42822 > ProgressReport should include an accurate effective shuffle partition number > > > Key: SPARK-45084 > URL: https://issues.apache.org/jira/browse/SPARK-45084 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 3.4.2 >Reporter: Siying Dong >Priority: Minor > > Currently, there is a numShufflePartitions "metric" reported in > StateOperatorProgress part of the progress report. However, the number is > reported by aggregating executors so in the case of task retry or speculative > executor, the metric is higher than number of shuffle partitions for the > query plan. Number of shuffle partitions can be useful for reporting purpose > so having a metric is helpful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true
[
https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruce Robbins updated SPARK-44805:
--
Affects Version/s: 3.4.1
> Data lost after union using
> spark.sql.parquet.enableNestedColumnVectorizedReader=true
> -
>
> Key: SPARK-44805
> URL: https://issues.apache.org/jira/browse/SPARK-44805
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.1, 3.4.1
> Environment: pySpark, linux, hadoop, parquet.
>Reporter: Jakub Wozniak
>Priority: Major
> Labels: correctness
>
> When union-ing two DataFrames read from parquet containing nested structures
> (2 fields of array types where one is double and second is integer) data from
> the second field seems to be lost (zeros are set instead).
> This seems to be the case only if nested vectorised reader is used
> (spark.sql.parquet.enableNestedColumnVectorizedReader=true).
> The following Python code reproduces the problem:
> {code:java}
> from pyspark.sql import SparkSession
> from pyspark.sql.types import *
> # PREPARING DATA
> data1 = []
> data2 = []
> for i in range(2):
> data1.append( (([1,2,3],[1,1,2]),i))
> data2.append( (([1.0,2.0,3.0],[1,1]),i+10))
> schema1 = StructType([
> StructField('value', StructType([
> StructField('f1', ArrayType(IntegerType()), True),
> StructField('f2', ArrayType(IntegerType()), True)
> ])),
> StructField('id', IntegerType(), True)
> ])
> schema2 = StructType([
> StructField('value', StructType([
> StructField('f1', ArrayType(DoubleType()), True),
> StructField('f2', ArrayType(IntegerType()), True)
> ])),
> StructField('id', IntegerType(), True)
> ])
> spark = SparkSession.builder.getOrCreate()
> data_dir = "/user//"
> df1 = spark.createDataFrame(data1, schema1)
> df1.write.mode('overwrite').parquet(data_dir + "data1")
> df2 = spark.createDataFrame(data2, schema2)
> df2.write.mode('overwrite').parquet(data_dir + "data2")
> # READING DATA
> parquet1 = spark.read.parquet(data_dir + "data1")
> parquet2 = spark.read.parquet(data_dir + "data2")
> # UNION
> out = parquet1.union(parquet2)
> parquet1.select("value.f2").distinct().show()
> out.select("value.f2").distinct().show()
> print(parquet1.collect())
> print(out.collect()) {code}
> Output:
> {code:java}
> +-+
> | f2|
> +-+
> |[1, 1, 2]|
> +-+
> +-+
> | f2|
> +-+
> |[0, 0, 0]|
> | [1, 1]|
> +-+
> [
> Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=0),
> Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=1)
> ]
> [
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=0),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=1),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=10),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=11)
> ] {code}
> Please notice that values for the field f2 are lost after the union is done.
> This only happens when this data is read from parquet files.
> Could you please look into this?
> Best regards,
> Jakub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45102) Support keyword columns on filters that interact with HMS
Steve Carlin created SPARK-45102: Summary: Support keyword columns on filters that interact with HMS Key: SPARK-45102 URL: https://issues.apache.org/jira/browse/SPARK-45102 Project: Spark Issue Type: Improvement Components: Connect Affects Versions: 3.4.1 Reporter: Steve Carlin Recently, https://issues.apache.org/jira/browse/HIVE-27665 was pushed on Hive. This will allow HMS to handle columns that are surrounded by backticks in filters. An example of a customer who hit this problem had a filter in Spark like this: where date='2015-01-06' This didn't work because the word "date" is a keyword. In order for the customer to work, the where clause should be changed to this: where `date`='2015-01-06' Spark strips out the backticks before passing the filter to HMS. We need to no longer strip the backticks as a configurable flag. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true
[
https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762792#comment-17762792
]
Bruce Robbins commented on SPARK-44805:
---
PR here: https://github.com/apache/spark/pull/42850
> Data lost after union using
> spark.sql.parquet.enableNestedColumnVectorizedReader=true
> -
>
> Key: SPARK-44805
> URL: https://issues.apache.org/jira/browse/SPARK-44805
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.1
> Environment: pySpark, linux, hadoop, parquet.
>Reporter: Jakub Wozniak
>Priority: Major
> Labels: correctness
>
> When union-ing two DataFrames read from parquet containing nested structures
> (2 fields of array types where one is double and second is integer) data from
> the second field seems to be lost (zeros are set instead).
> This seems to be the case only if nested vectorised reader is used
> (spark.sql.parquet.enableNestedColumnVectorizedReader=true).
> The following Python code reproduces the problem:
> {code:java}
> from pyspark.sql import SparkSession
> from pyspark.sql.types import *
> # PREPARING DATA
> data1 = []
> data2 = []
> for i in range(2):
> data1.append( (([1,2,3],[1,1,2]),i))
> data2.append( (([1.0,2.0,3.0],[1,1]),i+10))
> schema1 = StructType([
> StructField('value', StructType([
> StructField('f1', ArrayType(IntegerType()), True),
> StructField('f2', ArrayType(IntegerType()), True)
> ])),
> StructField('id', IntegerType(), True)
> ])
> schema2 = StructType([
> StructField('value', StructType([
> StructField('f1', ArrayType(DoubleType()), True),
> StructField('f2', ArrayType(IntegerType()), True)
> ])),
> StructField('id', IntegerType(), True)
> ])
> spark = SparkSession.builder.getOrCreate()
> data_dir = "/user//"
> df1 = spark.createDataFrame(data1, schema1)
> df1.write.mode('overwrite').parquet(data_dir + "data1")
> df2 = spark.createDataFrame(data2, schema2)
> df2.write.mode('overwrite').parquet(data_dir + "data2")
> # READING DATA
> parquet1 = spark.read.parquet(data_dir + "data1")
> parquet2 = spark.read.parquet(data_dir + "data2")
> # UNION
> out = parquet1.union(parquet2)
> parquet1.select("value.f2").distinct().show()
> out.select("value.f2").distinct().show()
> print(parquet1.collect())
> print(out.collect()) {code}
> Output:
> {code:java}
> +-+
> | f2|
> +-+
> |[1, 1, 2]|
> +-+
> +-+
> | f2|
> +-+
> |[0, 0, 0]|
> | [1, 1]|
> +-+
> [
> Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=0),
> Row(value=Row(f1=[1, 2, 3], f2=[1, 1, 2]), id=1)
> ]
> [
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=0),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[0, 0, 0]), id=1),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=10),
> Row(value=Row(f1=[1.0, 2.0, 3.0], f2=[1, 1]), id=11)
> ] {code}
> Please notice that values for the field f2 are lost after the union is done.
> This only happens when this data is read from parquet files.
> Could you please look into this?
> Best regards,
> Jakub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45101) Spark UI: A stage is still active even when all of it's tasks are succeeded
[ https://issues.apache.org/jira/browse/SPARK-45101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RickyMa updated SPARK-45101: Summary: Spark UI: A stage is still active even when all of it's tasks are succeeded (was: Spark UI: A stage is still active even all of it's tasks are succeeded) > Spark UI: A stage is still active even when all of it's tasks are succeeded > --- > > Key: SPARK-45101 > URL: https://issues.apache.org/jira/browse/SPARK-45101 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: RickyMa >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > In the stage UI, we can see all the tasks' statuses are SUCCESS. > But the stage is still marked as active. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45077) Upgrade dagre-d3.js from 0.4.3 to 0.6.4
[ https://issues.apache.org/jira/browse/SPARK-45077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45077. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42815 [https://github.com/apache/spark/pull/42815] > Upgrade dagre-d3.js from 0.4.3 to 0.6.4 > --- > > Key: SPARK-45077 > URL: https://issues.apache.org/jira/browse/SPARK-45077 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45101) Spark UI: A stage is still active even all of it's tasks are succeeded
[ https://issues.apache.org/jira/browse/SPARK-45101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RickyMa updated SPARK-45101: Description: In the stage UI, we can see all the tasks' statuses are SUCCESS. But the stage is still marked as active. was: Go into the stage UI, we can see all the tasks' statuses are SUCCESS. But the stage is still marked as active. > Spark UI: A stage is still active even all of it's tasks are succeeded > -- > > Key: SPARK-45101 > URL: https://issues.apache.org/jira/browse/SPARK-45101 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: RickyMa >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > In the stage UI, we can see all the tasks' statuses are SUCCESS. > But the stage is still marked as active. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45077) Upgrade dagre-d3.js from 0.4.3 to 0.6.4
[ https://issues.apache.org/jira/browse/SPARK-45077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45077: - Priority: Minor (was: Major) > Upgrade dagre-d3.js from 0.4.3 to 0.6.4 > --- > > Key: SPARK-45077 > URL: https://issues.apache.org/jira/browse/SPARK-45077 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Minor > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45101) Spark UI: A stage is still active even all of it's tasks are succeeded
[ https://issues.apache.org/jira/browse/SPARK-45101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RickyMa updated SPARK-45101: Description: Go into the stage UI, we can see all the tasks' statuses are SUCCESS. But the stage is still marked as active. was: A task is already killed. But it is still marked as succeeded, which causes the stage never stops. Go into the stage UI, we can see all the tasks' statuses are SUCCESS. > Spark UI: A stage is still active even all of it's tasks are succeeded > -- > > Key: SPARK-45101 > URL: https://issues.apache.org/jira/browse/SPARK-45101 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: RickyMa >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > Go into the stage UI, we can see all the tasks' statuses are SUCCESS. > But the stage is still marked as active. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45077) Upgrade dagre-d3.js from 0.4.3 to 0.6.4
[ https://issues.apache.org/jira/browse/SPARK-45077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45077: Assignee: Kent Yao > Upgrade dagre-d3.js from 0.4.3 to 0.6.4 > --- > > Key: SPARK-45077 > URL: https://issues.apache.org/jira/browse/SPARK-45077 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45101) Spark UI: A stage is still active even all of it's tasks are succeeded
[ https://issues.apache.org/jira/browse/SPARK-45101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RickyMa updated SPARK-45101: Summary: Spark UI: A stage is still active even all of it's tasks are succeeded (was: Spark UI: A killed task should not be marked as succeeded) > Spark UI: A stage is still active even all of it's tasks are succeeded > -- > > Key: SPARK-45101 > URL: https://issues.apache.org/jira/browse/SPARK-45101 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: RickyMa >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > A task is already killed. But it is still marked as succeeded, which causes > the stage never stops. > Go into the stage UI, we can see all the tasks' statuses are SUCCESS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45101) Spark UI: A killed task should not be marked as succeeded
[ https://issues.apache.org/jira/browse/SPARK-45101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RickyMa updated SPARK-45101: Attachment: 2.png 1.png 3.png > Spark UI: A killed task should not be marked as succeeded > - > > Key: SPARK-45101 > URL: https://issues.apache.org/jira/browse/SPARK-45101 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: RickyMa >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > A task is already killed. But it is still marked as succeeded, which causes > the stage never stops. > Go into the stage UI, we can see all the tasks' statuses are SUCCESS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45101) Spark UI: A killed task should not be marked as succeeded
[ https://issues.apache.org/jira/browse/SPARK-45101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RickyMa updated SPARK-45101: Description: A task is already killed. But it is still marked as succeeded, which causes the stage never stops. Go into the stage UI, we can see all the tasks' statuses are SUCCESS. was: A task is already killed. But it is still marked as succeeded, which causes the stage never stops. Go into the stage UI, we can see all the tasks' statuses are SUCCESS. !image-2023-09-07-21-45-08-667.png! !image-2023-09-07-21-44-39-394.png! !image-2023-09-07-21-59-54-033.png! !image-2023-09-07-21-59-50-002.png! > Spark UI: A killed task should not be marked as succeeded > - > > Key: SPARK-45101 > URL: https://issues.apache.org/jira/browse/SPARK-45101 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: RickyMa >Priority: Major > > A task is already killed. But it is still marked as succeeded, which causes > the stage never stops. > Go into the stage UI, we can see all the tasks' statuses are SUCCESS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45101) Spark UI: A killed task should not be marked as succeeded
RickyMa created SPARK-45101: --- Summary: Spark UI: A killed task should not be marked as succeeded Key: SPARK-45101 URL: https://issues.apache.org/jira/browse/SPARK-45101 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.4.1 Reporter: RickyMa A task is already killed. But it is still marked as succeeded, which causes the stage never stops. Go into the stage UI, we can see all the tasks' statuses are SUCCESS. !image-2023-09-07-21-45-08-667.png! !image-2023-09-07-21-44-39-394.png! !image-2023-09-07-21-59-54-033.png! !image-2023-09-07-21-59-50-002.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45100) reflect() fails with an internal error on NULL class and method
[
https://issues.apache.org/jira/browse/SPARK-45100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-45100:
-
Description:
The example below demonstrates the issue:
{code:sql}
spark-sql (default)> select reflect('java.util.UUID', CAST(NULL AS STRING));
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error.
You hit a bug in Spark or the Spark plugins you use. Please, report this bug to
the corresponding communities or vendors, and provide the full stack trace.
{code}
was:
The example below demonstrates the issue:
{code:sql}
spark-sql (default)> SELECT percentile_approx(col, array(0.5, 0.4, 0.1), NULL)
FROM VALUES (0), (1), (2), (10) AS tab(col);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error.
You hit a bug in Spark or the Spark plugins you use. Please, report this bug to
the corresponding communities or vendors, and provide the full stack trace.
{code}
> reflect() fails with an internal error on NULL class and method
> ---
>
> Key: SPARK-45100
> URL: https://issues.apache.org/jira/browse/SPARK-45100
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.3.2, 3.4.1, 3.5.0, 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Major
>
> The example below demonstrates the issue:
> {code:sql}
> spark-sql (default)> select reflect('java.util.UUID', CAST(NULL AS STRING));
> [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error.
> You hit a bug in Spark or the Spark plugins you use. Please, report this bug
> to the corresponding communities or vendors, and provide the full stack trace.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45100) reflect() fails with an internal error on NULL class and method
Max Gekk created SPARK-45100:
Summary: reflect() fails with an internal error on NULL class and
method
Key: SPARK-45100
URL: https://issues.apache.org/jira/browse/SPARK-45100
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.3.2, 3.4.1, 3.5.0, 4.0.0
Reporter: Max Gekk
Assignee: Max Gekk
Fix For: 3.4.2, 3.5.0, 4.0.0, 3.3.4
The example below demonstrates the issue:
{code:sql}
spark-sql (default)> SELECT percentile_approx(col, array(0.5, 0.4, 0.1), NULL)
FROM VALUES (0), (1), (2), (10) AS tab(col);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error.
You hit a bug in Spark or the Spark plugins you use. Please, report this bug to
the corresponding communities or vendors, and provide the full stack trace.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45100) reflect() fails with an internal error on NULL class and method
[
https://issues.apache.org/jira/browse/SPARK-45100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-45100:
-
Fix Version/s: (was: 3.5.0)
(was: 4.0.0)
(was: 3.4.2)
(was: 3.3.4)
> reflect() fails with an internal error on NULL class and method
> ---
>
> Key: SPARK-45100
> URL: https://issues.apache.org/jira/browse/SPARK-45100
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.3.2, 3.4.1, 3.5.0, 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Major
>
> The example below demonstrates the issue:
> {code:sql}
> spark-sql (default)> SELECT percentile_approx(col, array(0.5, 0.4, 0.1),
> NULL) FROM VALUES (0), (1), (2), (10) AS tab(col);
> [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error.
> You hit a bug in Spark or the Spark plugins you use. Please, report this bug
> to the corresponding communities or vendors, and provide the full stack trace.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Description:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns =
Seq("a", "b"), joinType = "outer"))
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
(Forgive the parens instead of brackets the code snippet please, Jira was
inferring macros)
was:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns =
Seq("a", "b"), joinType = "outer"))
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns
> = Seq("a", "b"), joinType = "outer"))
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
> (Forgive the parens instead of brackets the code snippet please, Jira was
> inferring macros)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Description:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft {
case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType
= "outer")
}{{{}{}}}{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
was:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}{{frames.reduceLeft }}{{{}{
case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType
= "outer")
}{}}}{{{}{}}}{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft {
> case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"),
> joinType = "outer")
> }{{{}{}}}{quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Description:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns =
Seq("a", "b"), joinType = "outer"))
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
was:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft {
case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType
= "outer")
}{{{}{}}}{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns
> = Seq("a", "b"), joinType = "outer"))
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Description:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}{{frames.reduceLeft }}{{{}{
case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType
= "outer")
}{}}}{{{}{}}}{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
was:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft({ case (l, r) => l.join(r.hint("merge"),
usingColumns = Seq("a", "b"), joinType = "outer") })
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}{{frames.reduceLeft }}{{{}{
> case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"),
> joinType = "outer")
> }{}}}{{{}{}}}{quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Description:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft({ case (l, r) => l.join(r.hint("merge"),
usingColumns = Seq("a", "b"), joinType = "outer") })
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
was:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft { case (l, r) => l.join(r.hint("merge"),
usingColumns = Seq("a", "b"), joinType = "outer") }
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft({ case (l, r) => l.join(r.hint("merge"),
> usingColumns = Seq("a", "b"), joinType = "outer") })
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Description:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft { case (l, r) => l.join(r.hint("merge"),
usingColumns = Seq("a", "b"), joinType = "outer") }
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
was:
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft { case (l, r) =>
l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType = "outer")
}
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft { case (l, r) => l.join(r.hint("merge"),
> usingColumns = Seq("a", "b"), joinType = "outer") }
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huw updated SPARK-45099:
Priority: Minor (was: Major)
> SortMergeExec with Outer using join forgets sort information
>
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1
>Reporter: Huw
>Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft { case (l, r) =>
> l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType = "outer")
> }
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45099) SortMergeExec with Outer using join forgets sort information
Huw created SPARK-45099:
---
Summary: SortMergeExec with Outer using join forgets sort
information
Key: SPARK-45099
URL: https://issues.apache.org/jira/browse/SPARK-45099
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.1
Reporter: Huw
When performing a 'using' join with a sort hint in a full outer, the
ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output
ordering for a full outer join as empty, even though these join pairs in their
final coalesced form actually are ordered.
This means that code like this:
{quote}frames.reduceLeft { case (l, r) =>
l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType = "outer")
}
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as
each join forgets its sort order.
Ideally this whole operation wouldn't require any shuffles if all the frames
are grouped and sorted by the keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes
[
https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762699#comment-17762699
]
Michael Negodaev commented on SPARK-34645:
--
Here is the dump: [^dump.txt].
> [K8S] Driver pod stuck in Running state after job completes
> ---
>
> Key: SPARK-34645
> URL: https://issues.apache.org/jira/browse/SPARK-34645
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.0.2
> Environment: Kubernetes:
> {code:java}
> Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2",
> GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean",
> BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc",
> Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5",
> GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean",
> BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc",
> Platform:"linux/amd64"}
> {code}
>Reporter: Andy Grove
>Priority: Major
> Attachments: dump.txt
>
>
> I am running automated benchmarks in k8s, using spark-submit in cluster mode,
> so the driver runs in a pod.
> When running with Spark 3.0.1 and 3.1.1 everything works as expected and I
> see the Spark context being shut down after the job completes.
> However, when running with Spark 3.0.2 I do not see the context get shut down
> and the driver pod is stuck in the Running state indefinitely.
> This is the output I see after job completion with 3.0.1 and 3.1.1 and this
> output does not appear with 3.0.2. With 3.0.2 there is no output at all after
> the job completes.
> {code:java}
> 2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped
> Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
> 2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at
> http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
> 2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting
> down all executors
> 2021-03-05 20:09:24,600 INFO
> k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
> executor to shut down
> 2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes
> client has been closed (this is expected if the application is shutting down.)
> 2021-03-05 20:09:24,719 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 2021-03-05 20:09:24,736 INFO memory.MemoryStore: MemoryStore cleared
> 2021-03-05 20:09:24,738 INFO storage.BlockManager: BlockManager stopped
> 2021-03-05 20:09:24,744 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 2021-03-05 20:09:24,752 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 2021-03-05 20:09:24,768 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 2021-03-05 20:09:24,768 INFO util.ShutdownHookManager: Shutdown hook called
> 2021-03-05 20:09:24,769 INFO util.ShutdownHookManager: Deleting directory
> /var/data/spark-67fa44df-e86c-463a-a149-25d95817ff8e/spark-a5476c14-c103-4108-b733-961400485d8a
> 2021-03-05 20:09:24,772 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-9d6261f5-4394-472b-9c9a-e22bde877814
> 2021-03-05 20:09:24,778 INFO impl.MetricsSystemImpl: Stopping s3a-file-system
> metrics system...
> 2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics
> system stopped.
> 2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics
> system shutdown complete.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes
[
https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Negodaev updated SPARK-34645:
-
Attachment: dump.txt
> [K8S] Driver pod stuck in Running state after job completes
> ---
>
> Key: SPARK-34645
> URL: https://issues.apache.org/jira/browse/SPARK-34645
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.0.2
> Environment: Kubernetes:
> {code:java}
> Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2",
> GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean",
> BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc",
> Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5",
> GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean",
> BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc",
> Platform:"linux/amd64"}
> {code}
>Reporter: Andy Grove
>Priority: Major
> Attachments: dump.txt
>
>
> I am running automated benchmarks in k8s, using spark-submit in cluster mode,
> so the driver runs in a pod.
> When running with Spark 3.0.1 and 3.1.1 everything works as expected and I
> see the Spark context being shut down after the job completes.
> However, when running with Spark 3.0.2 I do not see the context get shut down
> and the driver pod is stuck in the Running state indefinitely.
> This is the output I see after job completion with 3.0.1 and 3.1.1 and this
> output does not appear with 3.0.2. With 3.0.2 there is no output at all after
> the job completes.
> {code:java}
> 2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped
> Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
> 2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at
> http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
> 2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting
> down all executors
> 2021-03-05 20:09:24,600 INFO
> k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
> executor to shut down
> 2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes
> client has been closed (this is expected if the application is shutting down.)
> 2021-03-05 20:09:24,719 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 2021-03-05 20:09:24,736 INFO memory.MemoryStore: MemoryStore cleared
> 2021-03-05 20:09:24,738 INFO storage.BlockManager: BlockManager stopped
> 2021-03-05 20:09:24,744 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 2021-03-05 20:09:24,752 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 2021-03-05 20:09:24,768 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 2021-03-05 20:09:24,768 INFO util.ShutdownHookManager: Shutdown hook called
> 2021-03-05 20:09:24,769 INFO util.ShutdownHookManager: Deleting directory
> /var/data/spark-67fa44df-e86c-463a-a149-25d95817ff8e/spark-a5476c14-c103-4108-b733-961400485d8a
> 2021-03-05 20:09:24,772 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-9d6261f5-4394-472b-9c9a-e22bde877814
> 2021-03-05 20:09:24,778 INFO impl.MetricsSystemImpl: Stopping s3a-file-system
> metrics system...
> 2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics
> system stopped.
> 2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics
> system shutdown complete.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45080) Kafka DSv2 streaming source implementation calls planInputPartitions 4 times per microbatch
[ https://issues.apache.org/jira/browse/SPARK-45080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-45080: Assignee: Jungtaek Lim > Kafka DSv2 streaming source implementation calls planInputPartitions 4 times > per microbatch > --- > > Key: SPARK-45080 > URL: https://issues.apache.org/jira/browse/SPARK-45080 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > I was tracking through method calls for DSv2 streaming source, and figured > out planInputPartitions is called 4 times per microbatch. > It turned out that multiple calls of planInputPartitions is due to > `DataSourceV2ScanExecBase.supportsColumnar`, though it is called through > `MicroBatchScanExec.inputPartitions` which is defined as lazy, hence > shouldn't happen. > The behavior seems to be coupled with catalyst and very hard to figure out > why, but with SPARK-44505, we can at least fix this per each data source. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45080) Kafka DSv2 streaming source implementation calls planInputPartitions 4 times per microbatch
[ https://issues.apache.org/jira/browse/SPARK-45080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-45080. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42823 [https://github.com/apache/spark/pull/42823] > Kafka DSv2 streaming source implementation calls planInputPartitions 4 times > per microbatch > --- > > Key: SPARK-45080 > URL: https://issues.apache.org/jira/browse/SPARK-45080 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > Fix For: 4.0.0 > > > I was tracking through method calls for DSv2 streaming source, and figured > out planInputPartitions is called 4 times per microbatch. > It turned out that multiple calls of planInputPartitions is due to > `DataSourceV2ScanExecBase.supportsColumnar`, though it is called through > `MicroBatchScanExec.inputPartitions` which is defined as lazy, hence > shouldn't happen. > The behavior seems to be coupled with catalyst and very hard to figure out > why, but with SPARK-44505, we can at least fix this per each data source. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45098) Custom jekyll-rediect-from redirect.html template
Kent Yao created SPARK-45098: Summary: Custom jekyll-rediect-from redirect.html template Key: SPARK-45098 URL: https://issues.apache.org/jira/browse/SPARK-45098 Project: Spark Issue Type: Bug Components: Documentation Affects Versions: 3.5.0 Reporter: Kent Yao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes
[
https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762581#comment-17762581
]
Michael Negodaev edited comment on SPARK-34645 at 9/7/23 9:42 AM:
--
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster. When the job fails for some
reason driver terminates all the executors and stops the query and the context.
But the driver's pod get stuck into a Running status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103){code}
Noticed that the last messages of the driver log are:
{code:java}
INFO [org.apache.spark.SparkContext]: Successfully stopped SparkContext
Exception in thread "main"
org.apache.spark.sql.streaming.StreamingQueryException: ...
Driver stacktrace:
=== Streaming Query ===
... {code}
while in a regular situation ShutdownHookManager should be called and the last
messages should be
{code:java}
[shutdown-hook-0] INFO [org.apache.spark.util.ShutdownHookManager]: Shutdown
hook called
[shutdown-hook-0] INFO [org.apache.spark.util.ShutdownHookManager]: Deleting
directory /tmp/spark-1ba67a92-f374-4767-a233-54932195a2d5
...{code}
I can't see why ShutdownHookManager hasn't been called at the end of the job.
was (Author: mnegodaev):
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster. When the job fails for some
reason driver terminates all the executors and stops the query and the context.
But the driver's pod get stuck into a running status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103){code}
Noticed that the last messages of the driver log are:
{code:java}
INFO [org.apache.spark.SparkContext]: Successfully stopped SparkContext
Exception in thread "main"
org.apache.spark.sql.streaming.StreamingQueryException: ...
Driver stacktrace:
=== Streaming Query ===
... {code}
while in a regular situation ShutdownHookManager should be called and the last
messages should be
{code:java}
[shutdown-hook-0] INFO [org.apache.spark.util.ShutdownHookManager]: Shutdown
hook called
[shutdown-hook-0] INFO [org.apache.spark.util.ShutdownHookManager]: Deleting
directory /tmp/spark-1ba67a92-f374-4767-a233-54932195a2d5
...{code}
I can't see why ShutdownHookManager hasn't been called at the end of the job.
> [K8S] Driver pod stuck in Running state after job completes
> ---
>
> Key: SPARK-34645
> URL: https://issues.apache.org/jira/browse/SPARK-34645
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.0.2
> Environment: Kubernetes:
> {code:java}
> Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2",
> GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean",
> BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc",
> Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5",
> GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean",
> BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc",
> Platform:"linux/amd64"}
> {code}
>Reporter: Andy Grove
>Priority: Major
>
> I am running automated benchmarks in k8s, using spark-submit in cluster mode,
> so the driver runs in a pod.
> When running with Spark 3.0.1 and 3.1.1 everything works as expected and I
> see the Spark context being shut down after the job completes.
> However, when running with Spark 3.0.2 I do not see the context get shut down
> and the driver pod is stuck in the Running state indefinitely.
> This is the output I see after job completion with 3.0.1 and 3.1.1 and this
> output does not appear with 3.0.2. With 3.0.2 there is no output at all after
> the job completes.
> {code:java}
> 2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped
> Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
> 2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at
> http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
> 2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting
> down all executors
> 2021-03-05 20:09:24,600 INFO
> k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
> executor to shut down
> 2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes
> client has been closed (this
[jira] [Comment Edited] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes
[
https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762581#comment-17762581
]
Michael Negodaev edited comment on SPARK-34645 at 9/7/23 9:40 AM:
--
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster. When the job fails for some
reason driver terminates all the executors and stops the query and the context.
But the driver's pod get stuck into a running status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103){code}
Noticed that the last messages of the driver log are:
{code:java}
INFO [org.apache.spark.SparkContext]: Successfully stopped SparkContext
Exception in thread "main"
org.apache.spark.sql.streaming.StreamingQueryException: ...
Driver stacktrace:
=== Streaming Query ===
... {code}
while in a regular situation ShutdownHookManager should be called and the last
messages should be
{code:java}
[shutdown-hook-0] INFO [org.apache.spark.util.ShutdownHookManager]: Shutdown
hook called
[shutdown-hook-0] INFO [org.apache.spark.util.ShutdownHookManager]: Deleting
directory /tmp/spark-1ba67a92-f374-4767-a233-54932195a2d5
...{code}
I can't see why ShutdownHookManager hasn't been called at the end of the job.
was (Author: mnegodaev):
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster with
{code:java}
Trigger.ProcessingTime(duration){code}
When the job fails for some reason driver terminates all the executors and
stops the query and the context. But the driver's pod get stuck into a running
status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103)
{code}
I also noticed that changing the job trigger to
{code:java}
Trigger.Once() {code}
resolves the problem: after the job's failure k8s terminates driver's pod with
Error status.
> [K8S] Driver pod stuck in Running state after job completes
> ---
>
> Key: SPARK-34645
> URL: https://issues.apache.org/jira/browse/SPARK-34645
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.0.2
> Environment: Kubernetes:
> {code:java}
> Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2",
> GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean",
> BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc",
> Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5",
> GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean",
> BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc",
> Platform:"linux/amd64"}
> {code}
>Reporter: Andy Grove
>Priority: Major
>
> I am running automated benchmarks in k8s, using spark-submit in cluster mode,
> so the driver runs in a pod.
> When running with Spark 3.0.1 and 3.1.1 everything works as expected and I
> see the Spark context being shut down after the job completes.
> However, when running with Spark 3.0.2 I do not see the context get shut down
> and the driver pod is stuck in the Running state indefinitely.
> This is the output I see after job completion with 3.0.1 and 3.1.1 and this
> output does not appear with 3.0.2. With 3.0.2 there is no output at all after
> the job completes.
> {code:java}
> 2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped
> Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
> 2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at
> http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
> 2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting
> down all executors
> 2021-03-05 20:09:24,600 INFO
> k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
> executor to shut down
> 2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes
> client has been closed (this is expected if the application is shutting down.)
> 2021-03-05 20:09:24,719 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 2021-03-05 20:09:24,736 INFO memory.MemoryStore: MemoryStore cleared
> 2021-03-05 20:09:24,738 INFO storage.BlockManager: BlockManager stopped
> 2021-03-05 20:09:24,744 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 2021-03-05 20:09:24,752 INFO
>
[jira] [Resolved] (SPARK-45089) Remove obsolete repo of DB2 JDBC driver
[ https://issues.apache.org/jira/browse/SPARK-45089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-45089. - Fix Version/s: 4.0.0 Assignee: Cheng Pan Resolution: Fixed Issue resolved by pull request 42820 https://github.com/apache/spark/pull/42820 > Remove obsolete repo of DB2 JDBC driver > --- > > Key: SPARK-45089 > URL: https://issues.apache.org/jira/browse/SPARK-45089 > Project: Spark > Issue Type: Test > Components: Build, Tests >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Assignee: Cheng Pan >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45086) Display hexadecimal for thread lock hash code
[ https://issues.apache.org/jira/browse/SPARK-45086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-45086. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42826 [https://github.com/apache/spark/pull/42826] > Display hexadecimal for thread lock hash code > - > > Key: SPARK-45086 > URL: https://issues.apache.org/jira/browse/SPARK-45086 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.4.1, 3.5.0, 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45086) Display hexadecimal for thread lock hash code
[ https://issues.apache.org/jira/browse/SPARK-45086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-45086: Assignee: Kent Yao > Display hexadecimal for thread lock hash code > - > > Key: SPARK-45086 > URL: https://issues.apache.org/jira/browse/SPARK-45086 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.4.1, 3.5.0, 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45068) Make math function output column name displayed in upper-cased style
[ https://issues.apache.org/jira/browse/SPARK-45068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-45068: Summary: Make math function output column name displayed in upper-cased style (was: Make function output column name consistent in case) > Make math function output column name displayed in upper-cased style > > > Key: SPARK-45068 > URL: https://issues.apache.org/jira/browse/SPARK-45068 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes
[
https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762581#comment-17762581
]
Michael Negodaev edited comment on SPARK-34645 at 9/7/23 6:34 AM:
--
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster with
{code:java}
Trigger.ProcessingTime(duration){code}
When the job fails for some reason driver terminates all the executors and
stops the query and the context. But the driver's pod get stuck into a running
status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103)
{code}
I also noticed that changing the job trigger to
{code:java}
Trigger.Once() {code}
resolves the problem: after the job's failure k8s terminates driver's pod with
Error status.
was (Author: mnegodaev):
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster with
{code:java}
Trigger.ProcessingTime(duration){code}
When the job fails for some reason driver terminates all the executors and
stops the query and the context. But the driver's pod get stuck into a running
status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103)
{code}
I also noticed that changing the job trigger to
{code:java}
Trigger.Once() {code}
resolves the problem: after the job's failure k8s terminates driver's pod with
Error status.
> [K8S] Driver pod stuck in Running state after job completes
> ---
>
> Key: SPARK-34645
> URL: https://issues.apache.org/jira/browse/SPARK-34645
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.0.2
> Environment: Kubernetes:
> {code:java}
> Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2",
> GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean",
> BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc",
> Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5",
> GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean",
> BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc",
> Platform:"linux/amd64"}
> {code}
>Reporter: Andy Grove
>Priority: Major
>
> I am running automated benchmarks in k8s, using spark-submit in cluster mode,
> so the driver runs in a pod.
> When running with Spark 3.0.1 and 3.1.1 everything works as expected and I
> see the Spark context being shut down after the job completes.
> However, when running with Spark 3.0.2 I do not see the context get shut down
> and the driver pod is stuck in the Running state indefinitely.
> This is the output I see after job completion with 3.0.1 and 3.1.1 and this
> output does not appear with 3.0.2. With 3.0.2 there is no output at all after
> the job completes.
> {code:java}
> 2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped
> Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
> 2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at
> http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
> 2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting
> down all executors
> 2021-03-05 20:09:24,600 INFO
> k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
> executor to shut down
> 2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes
> client has been closed (this is expected if the application is shutting down.)
> 2021-03-05 20:09:24,719 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 2021-03-05 20:09:24,736 INFO memory.MemoryStore: MemoryStore cleared
> 2021-03-05 20:09:24,738 INFO storage.BlockManager: BlockManager stopped
> 2021-03-05 20:09:24,744 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 2021-03-05 20:09:24,752 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 2021-03-05 20:09:24,768 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 2021-03-05 20:09:24,768 INFO util.ShutdownHookManager: Shutdown hook called
> 2021-03-05 20:09:24,769 INFO util.ShutdownHookManager: Deleting directory
> /var/data/spark-67fa44df-e86c-463a-a149-25d95817ff8e/spark-a5476c14-c103-4108-b733-961400485d8a
> 2021-03-05 20:09:24,772 INFO util.ShutdownHookManager: Deleting
[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes
[
https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762581#comment-17762581
]
Michael Negodaev commented on SPARK-34645:
--
Faced with a similar problem with Spark 3.3.2 on k8s and JDK 11.
I'm running a streaming application on k8s cluster with
{code:java}
Trigger.ProcessingTime(duration){code}
When the job fails for some reason driver terminates all the executors and
stops the query and the context. But the driver's pod get stuck into a running
status.
Taking the dump shows that we're waiting for the driver's pod to complete:
{code:java}
at
org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl.watchOrStop(LoggingPodStatusWatcher.scala:103)
{code}
I also noticed that changing the job trigger to
{code:java}
Trigger.Once() {code}
resolves the problem: after the job's failure k8s terminates driver's pod with
Error status.
> [K8S] Driver pod stuck in Running state after job completes
> ---
>
> Key: SPARK-34645
> URL: https://issues.apache.org/jira/browse/SPARK-34645
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.0.2
> Environment: Kubernetes:
> {code:java}
> Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2",
> GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean",
> BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc",
> Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5",
> GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean",
> BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc",
> Platform:"linux/amd64"}
> {code}
>Reporter: Andy Grove
>Priority: Major
>
> I am running automated benchmarks in k8s, using spark-submit in cluster mode,
> so the driver runs in a pod.
> When running with Spark 3.0.1 and 3.1.1 everything works as expected and I
> see the Spark context being shut down after the job completes.
> However, when running with Spark 3.0.2 I do not see the context get shut down
> and the driver pod is stuck in the Running state indefinitely.
> This is the output I see after job completion with 3.0.1 and 3.1.1 and this
> output does not appear with 3.0.2. With 3.0.2 there is no output at all after
> the job completes.
> {code:java}
> 2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped
> Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
> 2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at
> http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
> 2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting
> down all executors
> 2021-03-05 20:09:24,600 INFO
> k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
> executor to shut down
> 2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes
> client has been closed (this is expected if the application is shutting down.)
> 2021-03-05 20:09:24,719 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 2021-03-05 20:09:24,736 INFO memory.MemoryStore: MemoryStore cleared
> 2021-03-05 20:09:24,738 INFO storage.BlockManager: BlockManager stopped
> 2021-03-05 20:09:24,744 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 2021-03-05 20:09:24,752 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 2021-03-05 20:09:24,768 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 2021-03-05 20:09:24,768 INFO util.ShutdownHookManager: Shutdown hook called
> 2021-03-05 20:09:24,769 INFO util.ShutdownHookManager: Deleting directory
> /var/data/spark-67fa44df-e86c-463a-a149-25d95817ff8e/spark-a5476c14-c103-4108-b733-961400485d8a
> 2021-03-05 20:09:24,772 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-9d6261f5-4394-472b-9c9a-e22bde877814
> 2021-03-05 20:09:24,778 INFO impl.MetricsSystemImpl: Stopping s3a-file-system
> metrics system...
> 2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics
> system stopped.
> 2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics
> system shutdown complete.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
