[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yikun Jiang updated SPARK-37759:
Description:
For Spark Test Runner:
{code:bash}
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 8U16G for Spark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
{code}
For PySpark Test Runner:
{code:bash}
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 4U8G for PySpark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64-pyspark
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
{code}
was:
{code:bash}
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 8U16G for Spark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64
# 4U8G for PySpark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64-pyspark
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
{code}
> Add ARM based self-hosted runner to Apache Spark
>
>
> Key: SPARK-37759
> URL: https://issues.apache.org/jira/browse/SPARK-37759
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yikun Jiang
>Priority: Major
>
> For Spark Test Runner:
> {code:bash}
> # Prepare the custom runner
> mkdir actions-runner && cd actions-runner
> curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
> https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> # Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
> # 8U16G for Spark
> ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
> ubuntu-20.04-arm64
> # Test on connecting
> ./run.sh
> # Start it as a service
> sudo ./svc.sh install
> sudo ./svc.sh start
> {code}
> For PySpark Test Runner:
> {code:bash}
> # Prepare the custom runner
> mkdir actions-runner && cd actions-runner
> curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
> https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> # Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
> # 4U8G for PySpark
> ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
> ubuntu-20.04-arm64-pyspark
> # Test on connecting
> ./run.sh
> # Start it as a service
> sudo ./svc.sh install
> sudo ./svc.sh start
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yikun Jiang updated SPARK-37759:
Description:
{code:bash}
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 8U16G for Spark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64
# 4U8G for PySpark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64-pyspark
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
{code}
was:
{code:shell}
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 8U16G for Spark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64
# 4U8G for PySpark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64-pyspark
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
{code}
> Add ARM based self-hosted runner to Apache Spark
>
>
> Key: SPARK-37759
> URL: https://issues.apache.org/jira/browse/SPARK-37759
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yikun Jiang
>Priority: Major
>
> {code:bash}
> # Prepare the custom runner
> mkdir actions-runner && cd actions-runner
> curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
> https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> # Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
> # 8U16G for Spark
> ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
> ubuntu-20.04-arm64
> # 4U8G for PySpark
> ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
> ubuntu-20.04-arm64-pyspark
> # Test on connecting
> ./run.sh
> # Start it as a service
> sudo ./svc.sh install
> sudo ./svc.sh start
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yikun Jiang updated SPARK-37759:
Description:
{code:shell}
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 8U16G for Spark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64
# 4U8G for PySpark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64-pyspark
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
{code}
was:
# Prepare the custom runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
# Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
# 8U16G for Spark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64
# 4U8G for PySpark
./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
ubuntu-20.04-arm64-pyspark
# Test on connecting
./run.sh
# Start it as a service
sudo ./svc.sh install
sudo ./svc.sh start
> Add ARM based self-hosted runner to Apache Spark
>
>
> Key: SPARK-37759
> URL: https://issues.apache.org/jira/browse/SPARK-37759
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yikun Jiang
>Priority: Major
>
> {code:shell}
> # Prepare the custom runner
> mkdir actions-runner && cd actions-runner
> curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L
> https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz
> # Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN
> # 8U16G for Spark
> ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
> ubuntu-20.04-arm64
> # 4U8G for PySpark
> ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels
> ubuntu-20.04-arm64-pyspark
> # Test on connecting
> ./run.sh
> # Start it as a service
> sudo ./svc.sh install
> sudo ./svc.sh start
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark
[ https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37759: Description: # Prepare the custom runner mkdir actions-runner && cd actions-runner curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz # Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN # 8U16G for Spark ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels ubuntu-20.04-arm64 # 4U8G for PySpark ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels ubuntu-20.04-arm64-pyspark # Test on connecting ./run.sh # Start it as a service sudo ./svc.sh install sudo ./svc.sh start > Add ARM based self-hosted runner to Apache Spark > > > Key: SPARK-37759 > URL: https://issues.apache.org/jira/browse/SPARK-37759 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.3.0 >Reporter: Yikun Jiang >Priority: Major > > # Prepare the custom runner > mkdir actions-runner && cd actions-runner > curl -o actions-runner-linux-arm64-2.286.0-airflow1.tar.gz -L > https://github.com/ashb/runner/releases/download/v2.286.0-airflow1/actions-runner-linux-arm64-2.286.0-airflow1.tar.gz > tar xzf ./actions-runner-linux-arm64-2.286.0-airflow1.tar.gz > # Configure the self-hosted runner, the Apache Infra Team will give a $TOKEN > # 8U16G for Spark > ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels > ubuntu-20.04-arm64 > # 4U8G for PySpark > ./config.sh --url https://github.com/apache/spark --token $TOKEN --labels > ubuntu-20.04-arm64-pyspark > # Test on connecting > ./run.sh > # Start it as a service > sudo ./svc.sh install > sudo ./svc.sh start -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37822) SQL function `split` should return an array of non-nullable elements
[
https://issues.apache.org/jira/browse/SPARK-37822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-37822.
-
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 35111
[https://github.com/apache/spark/pull/35111]
> SQL function `split` should return an array of non-nullable elements
>
>
> Key: SPARK-37822
> URL: https://issues.apache.org/jira/browse/SPARK-37822
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Shardul Mahadik
>Assignee: Shardul Mahadik
>Priority: Major
> Fix For: 3.3.0
>
>
> Currently, {{split}} [returns the data
> type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532]
> {{ArrayType(StringType)}} which means the resultant array can contain
> nullable elements. However I do not see any case where the array can contain
> nulls.
> In the case where either the provided string or delimiter is NULL, the output
> will be a NULL array. In case of empty string or no chars between delemiters,
> the output array will contain empty strings but never NULLs. So I propose we
> change the return type of {{split}} to mark elements as non-null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37822) SQL function `split` should return an array of non-nullable elements
[
https://issues.apache.org/jira/browse/SPARK-37822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-37822:
---
Assignee: Shardul Mahadik
> SQL function `split` should return an array of non-nullable elements
>
>
> Key: SPARK-37822
> URL: https://issues.apache.org/jira/browse/SPARK-37822
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Shardul Mahadik
>Assignee: Shardul Mahadik
>Priority: Major
>
> Currently, {{split}} [returns the data
> type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532]
> {{ArrayType(StringType)}} which means the resultant array can contain
> nullable elements. However I do not see any case where the array can contain
> nulls.
> In the case where either the provided string or delimiter is NULL, the output
> will be a NULL array. In case of empty string or no chars between delemiters,
> the output array will contain empty strings but never NULLs. So I propose we
> change the return type of {{split}} to mark elements as non-null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37825) Make spark beeline be able to handle javaOpts
[ https://issues.apache.org/jira/browse/SPARK-37825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37825: Assignee: Apache Spark > Make spark beeline be able to handle javaOpts > - > > Key: SPARK-37825 > URL: https://issues.apache.org/jira/browse/SPARK-37825 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Assignee: Apache Spark >Priority: Major > > Currently, we build the beeline command with SPARK_DRIVER_MEMORY only and are > not able to set extra java opts. Besides, the beeline is not a DRIVER-type > thing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37825) Make spark beeline be able to handle javaOpts
[ https://issues.apache.org/jira/browse/SPARK-37825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469747#comment-17469747 ] Apache Spark commented on SPARK-37825: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/35116 > Make spark beeline be able to handle javaOpts > - > > Key: SPARK-37825 > URL: https://issues.apache.org/jira/browse/SPARK-37825 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Currently, we build the beeline command with SPARK_DRIVER_MEMORY only and are > not able to set extra java opts. Besides, the beeline is not a DRIVER-type > thing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37825) Make spark beeline be able to handle javaOpts
[ https://issues.apache.org/jira/browse/SPARK-37825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37825: Assignee: (was: Apache Spark) > Make spark beeline be able to handle javaOpts > - > > Key: SPARK-37825 > URL: https://issues.apache.org/jira/browse/SPARK-37825 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Currently, we build the beeline command with SPARK_DRIVER_MEMORY only and are > not able to set extra java opts. Besides, the beeline is not a DRIVER-type > thing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37810) Executor Rolling in Kubernetes environment
[ https://issues.apache.org/jira/browse/SPARK-37810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37810: -- Labels: releasenotes (was: ) > Executor Rolling in Kubernetes environment > -- > > Key: SPARK-37810 > URL: https://issues.apache.org/jira/browse/SPARK-37810 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: releasenotes > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-37028) Add a 'kill' executor link in the Web UI.
[ https://issues.apache.org/jira/browse/SPARK-37028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-37028. - > Add a 'kill' executor link in the Web UI. > -- > > Key: SPARK-37028 > URL: https://issues.apache.org/jira/browse/SPARK-37028 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: weixiuli >Priority: Major > > The executor which is running in a bad node(eg. The system is overloaded or > disks are busy) or has big GC overheads may affect the efficiency of job > execution, although there are speculative mechanisms to resolve this problem, > but sometimes the speculated task may also run in a bad executor. > We should have a 'kill' link for each executor, similar to what we have for > each stage, so it's easier for users to kill executors in the UI. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37028) Add a 'kill' executor link in the Web UI.
[ https://issues.apache.org/jira/browse/SPARK-37028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37028. --- Resolution: Won't Do > Add a 'kill' executor link in the Web UI. > -- > > Key: SPARK-37028 > URL: https://issues.apache.org/jira/browse/SPARK-37028 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: weixiuli >Priority: Major > > The executor which is running in a bad node(eg. The system is overloaded or > disks are busy) or has big GC overheads may affect the efficiency of job > execution, although there are speculative mechanisms to resolve this problem, > but sometimes the speculated task may also run in a bad executor. > We should have a 'kill' link for each executor, similar to what we have for > each stage, so it's easier for users to kill executors in the UI. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37824) Document K8s executor rolling configurations
[ https://issues.apache.org/jira/browse/SPARK-37824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37824. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35114 [https://github.com/apache/spark/pull/35114] > Document K8s executor rolling configurations > > > Key: SPARK-37824 > URL: https://issues.apache.org/jira/browse/SPARK-37824 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37824) Document K8s executor rolling configurations
[ https://issues.apache.org/jira/browse/SPARK-37824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37824: - Assignee: Dongjoon Hyun > Document K8s executor rolling configurations > > > Key: SPARK-37824 > URL: https://issues.apache.org/jira/browse/SPARK-37824 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37787) Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not found in cache Exception
[ https://issues.apache.org/jira/browse/SPARK-37787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469712#comment-17469712 ] Apache Spark commented on SPARK-37787: -- User 'huangzhir' has created a pull request for this issue: https://github.com/apache/spark/pull/35115 > Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not > found in cache Exception > - > > Key: SPARK-37787 > URL: https://issues.apache.org/jira/browse/SPARK-37787 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0, 3.1.0, 3.2.0 > Environment: spark3 thrift server > > spark-default.conf > spark.hadoop.fs.hdfs.impl.disable.cache=true > >Reporter: huangzhir >Priority: Major > Original Estimate: 168h > Remaining Estimate: 168h > > *HDFS_DELEGATE_TOKEN not found in cache exception* occurs when accessing > spark thriftserver service. The specific exception is as follows: > [Exception Log | > https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png! > the HadoopDelegationTokenManager thow Exception when renewal > DelegationToken,as follows: > > We are also find HadoopDelegationTokenManager log as follows: > INFO [Credential Renewal Thread] > org.apache.spark.deploy.security.HadoopDelegationTokenManager logInfo - > *Scheduling renewal in 1921535501304.2 h.* > [hdfs Exceptin log in HadoopDelegationTokenManager > |https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37787) Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not found in cache Exception
[ https://issues.apache.org/jira/browse/SPARK-37787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37787: Assignee: Apache Spark > Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not > found in cache Exception > - > > Key: SPARK-37787 > URL: https://issues.apache.org/jira/browse/SPARK-37787 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0, 3.1.0, 3.2.0 > Environment: spark3 thrift server > > spark-default.conf > spark.hadoop.fs.hdfs.impl.disable.cache=true > >Reporter: huangzhir >Assignee: Apache Spark >Priority: Major > Original Estimate: 168h > Remaining Estimate: 168h > > *HDFS_DELEGATE_TOKEN not found in cache exception* occurs when accessing > spark thriftserver service. The specific exception is as follows: > [Exception Log | > https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png! > the HadoopDelegationTokenManager thow Exception when renewal > DelegationToken,as follows: > > We are also find HadoopDelegationTokenManager log as follows: > INFO [Credential Renewal Thread] > org.apache.spark.deploy.security.HadoopDelegationTokenManager logInfo - > *Scheduling renewal in 1921535501304.2 h.* > [hdfs Exceptin log in HadoopDelegationTokenManager > |https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37787) Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not found in cache Exception
[ https://issues.apache.org/jira/browse/SPARK-37787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469707#comment-17469707 ] Apache Spark commented on SPARK-37787: -- User 'huangzhir' has created a pull request for this issue: https://github.com/apache/spark/pull/35115 > Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not > found in cache Exception > - > > Key: SPARK-37787 > URL: https://issues.apache.org/jira/browse/SPARK-37787 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0, 3.1.0, 3.2.0 > Environment: spark3 thrift server > > spark-default.conf > spark.hadoop.fs.hdfs.impl.disable.cache=true > >Reporter: huangzhir >Priority: Major > Original Estimate: 168h > Remaining Estimate: 168h > > *HDFS_DELEGATE_TOKEN not found in cache exception* occurs when accessing > spark thriftserver service. The specific exception is as follows: > [Exception Log | > https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png! > the HadoopDelegationTokenManager thow Exception when renewal > DelegationToken,as follows: > > We are also find HadoopDelegationTokenManager log as follows: > INFO [Credential Renewal Thread] > org.apache.spark.deploy.security.HadoopDelegationTokenManager logInfo - > *Scheduling renewal in 1921535501304.2 h.* > [hdfs Exceptin log in HadoopDelegationTokenManager > |https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37787) Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not found in cache Exception
[ https://issues.apache.org/jira/browse/SPARK-37787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37787: Assignee: (was: Apache Spark) > Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not > found in cache Exception > - > > Key: SPARK-37787 > URL: https://issues.apache.org/jira/browse/SPARK-37787 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0, 3.1.0, 3.2.0 > Environment: spark3 thrift server > > spark-default.conf > spark.hadoop.fs.hdfs.impl.disable.cache=true > >Reporter: huangzhir >Priority: Major > Original Estimate: 168h > Remaining Estimate: 168h > > *HDFS_DELEGATE_TOKEN not found in cache exception* occurs when accessing > spark thriftserver service. The specific exception is as follows: > [Exception Log | > https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png! > the HadoopDelegationTokenManager thow Exception when renewal > DelegationToken,as follows: > > We are also find HadoopDelegationTokenManager log as follows: > INFO [Credential Renewal Thread] > org.apache.spark.deploy.security.HadoopDelegationTokenManager logInfo - > *Scheduling renewal in 1921535501304.2 h.* > [hdfs Exceptin log in HadoopDelegationTokenManager > |https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png] > !https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37825) Make spark beeline be able to handle javaOpts
[ https://issues.apache.org/jira/browse/SPARK-37825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-37825: - Issue Type: Improvement (was: Bug) > Make spark beeline be able to handle javaOpts > - > > Key: SPARK-37825 > URL: https://issues.apache.org/jira/browse/SPARK-37825 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Currently, we build the beeline command with SPARK_DRIVER_MEMORY only and are > not able to set extra java opts. Besides, the beeline is not a DRIVER-type > thing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37825) Make spark beeline be able to handle javaOpts
Kent Yao created SPARK-37825: Summary: Make spark beeline be able to handle javaOpts Key: SPARK-37825 URL: https://issues.apache.org/jira/browse/SPARK-37825 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.0 Reporter: Kent Yao Currently, we build the beeline command with SPARK_DRIVER_MEMORY only and are not able to set extra java opts. Besides, the beeline is not a DRIVER-type thing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37824) Document K8s executor rolling configurations
[ https://issues.apache.org/jira/browse/SPARK-37824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469685#comment-17469685 ] Apache Spark commented on SPARK-37824: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35114 > Document K8s executor rolling configurations > > > Key: SPARK-37824 > URL: https://issues.apache.org/jira/browse/SPARK-37824 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37824) Document K8s executor rolling configurations
[ https://issues.apache.org/jira/browse/SPARK-37824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37824: Assignee: Apache Spark > Document K8s executor rolling configurations > > > Key: SPARK-37824 > URL: https://issues.apache.org/jira/browse/SPARK-37824 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37824) Document K8s executor rolling configurations
[ https://issues.apache.org/jira/browse/SPARK-37824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37824: Assignee: (was: Apache Spark) > Document K8s executor rolling configurations > > > Key: SPARK-37824 > URL: https://issues.apache.org/jira/browse/SPARK-37824 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37824) Document K8s executor rolling configurations
[ https://issues.apache.org/jira/browse/SPARK-37824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469684#comment-17469684 ] Apache Spark commented on SPARK-37824: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35114 > Document K8s executor rolling configurations > > > Key: SPARK-37824 > URL: https://issues.apache.org/jira/browse/SPARK-37824 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37824) Document K8s executor rolling configurations
Dongjoon Hyun created SPARK-37824: - Summary: Document K8s executor rolling configurations Key: SPARK-37824 URL: https://issues.apache.org/jira/browse/SPARK-37824 Project: Spark Issue Type: Sub-task Components: Documentation Affects Versions: 3.3.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37636) Migrate CREATE NAMESPACE to use v2 command by default
[ https://issues.apache.org/jira/browse/SPARK-37636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469682#comment-17469682 ] Apache Spark commented on SPARK-37636: -- User 'imback82' has created a pull request for this issue: https://github.com/apache/spark/pull/35113 > Migrate CREATE NAMESPACE to use v2 command by default > - > > Key: SPARK-37636 > URL: https://issues.apache.org/jira/browse/SPARK-37636 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Terry Kim >Priority: Major > > Migrate CREATE NAMESPACE to use v2 command by default -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37636) Migrate CREATE NAMESPACE to use v2 command by default
[ https://issues.apache.org/jira/browse/SPARK-37636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469681#comment-17469681 ] Apache Spark commented on SPARK-37636: -- User 'imback82' has created a pull request for this issue: https://github.com/apache/spark/pull/35113 > Migrate CREATE NAMESPACE to use v2 command by default > - > > Key: SPARK-37636 > URL: https://issues.apache.org/jira/browse/SPARK-37636 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Terry Kim >Priority: Major > > Migrate CREATE NAMESPACE to use v2 command by default -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37636) Migrate CREATE NAMESPACE to use v2 command by default
[ https://issues.apache.org/jira/browse/SPARK-37636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37636: Assignee: Apache Spark > Migrate CREATE NAMESPACE to use v2 command by default > - > > Key: SPARK-37636 > URL: https://issues.apache.org/jira/browse/SPARK-37636 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Terry Kim >Assignee: Apache Spark >Priority: Major > > Migrate CREATE NAMESPACE to use v2 command by default -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37636) Migrate CREATE NAMESPACE to use v2 command by default
[ https://issues.apache.org/jira/browse/SPARK-37636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37636: Assignee: (was: Apache Spark) > Migrate CREATE NAMESPACE to use v2 command by default > - > > Key: SPARK-37636 > URL: https://issues.apache.org/jira/browse/SPARK-37636 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Terry Kim >Priority: Major > > Migrate CREATE NAMESPACE to use v2 command by default -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
[ https://issues.apache.org/jira/browse/SPARK-37819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37819. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35109 [https://github.com/apache/spark/pull/35109] > Add OUTLIER executor roll policy and use it by default > -- > > Key: SPARK-37819 > URL: https://issues.apache.org/jira/browse/SPARK-37819 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37823) Add `is-changed.py` dev script
[ https://issues.apache.org/jira/browse/SPARK-37823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469666#comment-17469666 ] Apache Spark commented on SPARK-37823: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35112 > Add `is-changed.py` dev script > -- > > Key: SPARK-37823 > URL: https://issues.apache.org/jira/browse/SPARK-37823 > Project: Spark > Issue Type: Test > Components: Project Infra, Tests >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37823) Add `is-changed.py` dev script
[ https://issues.apache.org/jira/browse/SPARK-37823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37823: Assignee: (was: Apache Spark) > Add `is-changed.py` dev script > -- > > Key: SPARK-37823 > URL: https://issues.apache.org/jira/browse/SPARK-37823 > Project: Spark > Issue Type: Test > Components: Project Infra, Tests >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37823) Add `is-changed.py` dev script
[ https://issues.apache.org/jira/browse/SPARK-37823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469664#comment-17469664 ] Apache Spark commented on SPARK-37823: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35112 > Add `is-changed.py` dev script > -- > > Key: SPARK-37823 > URL: https://issues.apache.org/jira/browse/SPARK-37823 > Project: Spark > Issue Type: Test > Components: Project Infra, Tests >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37823) Add `is-changed.py` dev script
[ https://issues.apache.org/jira/browse/SPARK-37823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37823: Assignee: Apache Spark > Add `is-changed.py` dev script > -- > > Key: SPARK-37823 > URL: https://issues.apache.org/jira/browse/SPARK-37823 > Project: Spark > Issue Type: Test > Components: Project Infra, Tests >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37823) Add `is-changed.py` dev script
Dongjoon Hyun created SPARK-37823: - Summary: Add `is-changed.py` dev script Key: SPARK-37823 URL: https://issues.apache.org/jira/browse/SPARK-37823 Project: Spark Issue Type: Test Components: Project Infra, Tests Affects Versions: 3.3.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37822) SQL function `split` should return an array of non-nullable elements
[
https://issues.apache.org/jira/browse/SPARK-37822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37822:
Assignee: (was: Apache Spark)
> SQL function `split` should return an array of non-nullable elements
>
>
> Key: SPARK-37822
> URL: https://issues.apache.org/jira/browse/SPARK-37822
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Shardul Mahadik
>Priority: Major
>
> Currently, {{split}} [returns the data
> type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532]
> {{ArrayType(StringType)}} which means the resultant array can contain
> nullable elements. However I do not see any case where the array can contain
> nulls.
> In the case where either the provided string or delimiter is NULL, the output
> will be a NULL array. In case of empty string or no chars between delemiters,
> the output array will contain empty strings but never NULLs. So I propose we
> change the return type of {{split}} to mark elements as non-null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37822) SQL function `split` should return an array of non-nullable elements
[
https://issues.apache.org/jira/browse/SPARK-37822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37822:
Assignee: Apache Spark
> SQL function `split` should return an array of non-nullable elements
>
>
> Key: SPARK-37822
> URL: https://issues.apache.org/jira/browse/SPARK-37822
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Shardul Mahadik
>Assignee: Apache Spark
>Priority: Major
>
> Currently, {{split}} [returns the data
> type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532]
> {{ArrayType(StringType)}} which means the resultant array can contain
> nullable elements. However I do not see any case where the array can contain
> nulls.
> In the case where either the provided string or delimiter is NULL, the output
> will be a NULL array. In case of empty string or no chars between delemiters,
> the output array will contain empty strings but never NULLs. So I propose we
> change the return type of {{split}} to mark elements as non-null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37822) SQL function `split` should return an array of non-nullable elements
[
https://issues.apache.org/jira/browse/SPARK-37822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469658#comment-17469658
]
Apache Spark commented on SPARK-37822:
--
User 'shardulm94' has created a pull request for this issue:
https://github.com/apache/spark/pull/35111
> SQL function `split` should return an array of non-nullable elements
>
>
> Key: SPARK-37822
> URL: https://issues.apache.org/jira/browse/SPARK-37822
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Shardul Mahadik
>Priority: Major
>
> Currently, {{split}} [returns the data
> type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532]
> {{ArrayType(StringType)}} which means the resultant array can contain
> nullable elements. However I do not see any case where the array can contain
> nulls.
> In the case where either the provided string or delimiter is NULL, the output
> will be a NULL array. In case of empty string or no chars between delemiters,
> the output array will contain empty strings but never NULLs. So I propose we
> change the return type of {{split}} to mark elements as non-null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37758) [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner
[
https://issues.apache.org/jira/browse/SPARK-37758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469654#comment-17469654
]
Yikun Jiang commented on SPARK-37758:
-
{code:java}
sudo apt install docker.io
sudo gpasswd -a $USER docker
newgrp docker
{code}
The pyspark runner is based on docker, the above cmd is the complete OS
packages depency.
> [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner
> --
>
> Key: SPARK-37758
> URL: https://issues.apache.org/jira/browse/SPARK-37758
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yikun Jiang
>Priority: Major
>
> cpython 3.9
> pypy3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner
[
https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469653#comment-17469653
]
Yikun Jiang edited comment on SPARK-37757 at 1/6/22, 3:34 AM:
--
{code:java}
sudo apt update
sudo apt-get install --no-install-recommends -y software-properties-common git
libxml2-dev pkg-config curl wget openjdk-8-jdk libpython3-dev python3-pip
python3-setuptools python3.8 python3.9 python-is-python3 libssl-dev
sudo update-alternatives --set java
/usr/lib/jvm/java-8-openjdk-arm64/jre/bin/java{code}
The above cmd is the complete os package dependecy for spark test runner.
was (Author: yikunkero):
{code:java}
sudo apt update sudo apt-get install --no-install-recommends -y
software-properties-common git libxml2-dev pkg-config curl wget openjdk-8-jdk
libpython3-dev python3-pip python3-setuptools python3.8 python3.9
python-is-python3 libssl-dev sudo update-alternatives --set java
/usr/lib/jvm/java-8-openjdk-arm64/jre/bin/java{code}
The above cmd is the complete os package dependecy for spark test runner.
> [Scala] Enable Spark test scheduled job on ARM based self-hosted runner
> ---
>
> Key: SPARK-37757
> URL: https://issues.apache.org/jira/browse/SPARK-37757
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yikun Jiang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner
[
https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469653#comment-17469653
]
Yikun Jiang commented on SPARK-37757:
-
{code:java}
sudo apt update sudo apt-get install --no-install-recommends -y
software-properties-common git libxml2-dev pkg-config curl wget openjdk-8-jdk
libpython3-dev python3-pip python3-setuptools python3.8 python3.9
python-is-python3 libssl-dev sudo update-alternatives --set java
/usr/lib/jvm/java-8-openjdk-arm64/jre/bin/java{code}
The above cmd is the complete os package dependecy for spark test runner.
> [Scala] Enable Spark test scheduled job on ARM based self-hosted runner
> ---
>
> Key: SPARK-37757
> URL: https://issues.apache.org/jira/browse/SPARK-37757
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yikun Jiang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37822) SQL function `split` should return an array of non-nullable elements
Shardul Mahadik created SPARK-37822:
---
Summary: SQL function `split` should return an array of
non-nullable elements
Key: SPARK-37822
URL: https://issues.apache.org/jira/browse/SPARK-37822
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: Shardul Mahadik
Currently, {{split}} [returns the data
type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532]
{{ArrayType(StringType)}} which means the resultant array can contain nullable
elements. However I do not see any case where the array can contain nulls.
In the case where either the provided string or delimiter is NULL, the output
will be a NULL array. In case of empty string or no chars between delemiters,
the output array will contain empty strings but never NULLs. So I propose we
change the return type of {{split}} to mark elements as non-null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469646#comment-17469646 ] Yikun Jiang commented on SPARK-37757: - `apt install libssl-dev`, then it's stable to passed, because need to upgrade openssl to 1.1.x, see related: https://issues.apache.org/jira/browse/CRYPTO-139 > [Scala] Enable Spark test scheduled job on ARM based self-hosted runner > --- > > Key: SPARK-37757 > URL: https://issues.apache.org/jira/browse/SPARK-37757 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.3.0 >Reporter: Yikun Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37820) Replace ApacheCommonBase64 with JavaBase64 for string fucntions
[ https://issues.apache.org/jira/browse/SPARK-37820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37820: Assignee: Apache Spark > Replace ApacheCommonBase64 with JavaBase64 for string fucntions > --- > > Key: SPARK-37820 > URL: https://issues.apache.org/jira/browse/SPARK-37820 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Assignee: Apache Spark >Priority: Major > > Replace dependency on third-party libraries with native > support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html > ) for Base-64 encode/decode. > 1. Performace gain > http://java-performance.info/base64-encoding-and-decoding-performance/ > 2. reduce dependencies afterward > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37820) Replace ApacheCommonBase64 with JavaBase64 for string fucntions
[ https://issues.apache.org/jira/browse/SPARK-37820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469643#comment-17469643 ] Apache Spark commented on SPARK-37820: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/35110 > Replace ApacheCommonBase64 with JavaBase64 for string fucntions > --- > > Key: SPARK-37820 > URL: https://issues.apache.org/jira/browse/SPARK-37820 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Replace dependency on third-party libraries with native > support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html > ) for Base-64 encode/decode. > 1. Performace gain > http://java-performance.info/base64-encoding-and-decoding-performance/ > 2. reduce dependencies afterward > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37820) Replace ApacheCommonBase64 with JavaBase64 for string fucntions
[ https://issues.apache.org/jira/browse/SPARK-37820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37820: Assignee: (was: Apache Spark) > Replace ApacheCommonBase64 with JavaBase64 for string fucntions > --- > > Key: SPARK-37820 > URL: https://issues.apache.org/jira/browse/SPARK-37820 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Replace dependency on third-party libraries with native > support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html > ) for Base-64 encode/decode. > 1. Performace gain > http://java-performance.info/base64-encoding-and-decoding-performance/ > 2. reduce dependencies afterward > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37821) spark thrift server RDD ID overflow lead sql execute failed
[ https://issues.apache.org/jira/browse/SPARK-37821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] muhong updated SPARK-37821: --- Description: this problem will happen in long run spark application，such as thrift server； as only one SparkContext instance in thrift server driver size，so if the concurrency of sql request is large or the sql is too complicate（this will create a lot of rdd）, the rdd will be generate too fast , the rdd id （SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala] ）will be consume fast, after a few months the nextRddId will overflow。the newRddId will be negative number，but the rdd's block id need to be positive, so this will lead a exception"Failed to parse rdd_-2123452330_2 into block ID"（rdd block id formate“val RDD = "rdd_([0-9]+)_([0-9]+)".r”：[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockId.scala]），so can not exchange data during sql execution, and lead sql execute failed was: this problem will happen in long run spark application，such as thrift server； as only one SparkContext instance in thrift server driver size，so if the concurrency of sql request is large or the sql is too complicate（this will create a lot of rdd）, the rdd will be generate too fast , the rdd id （SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala] ）will be consume fast, after a few months the nextRddId will overflow。the newRddId will be negative number，but the rdd's block id need to be positive, so this will lead a exception"Failed to parse rdd_-2123452330_2 into block ID"，so can not exchange data during sql execution, and lead sql execute failed > spark thrift server RDD ID overflow lead sql execute failed > --- > > Key: SPARK-37821 > URL: https://issues.apache.org/jira/browse/SPARK-37821 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: muhong >Priority: Major > > this problem will happen in long run spark application，such as thrift server； > as only one SparkContext instance in thrift server driver size，so if the > concurrency of sql request is large or the sql is too complicate（this will > create a lot of rdd）, the rdd will be generate too fast , the rdd id > （SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala] > ）will be consume fast, after a few months the nextRddId will overflow。the > newRddId will be negative number，but the rdd's block id need to be positive, > so this will lead a exception"Failed to parse rdd_-2123452330_2 into block > ID"（rdd block id formate“val RDD = > "rdd_([0-9]+)_([0-9]+)".r”：[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockId.scala]），so > can not exchange data during sql execution, and lead sql execute failed -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37821) spark thrift server RDD ID overflow lead sql execute failed
muhong created SPARK-37821: -- Summary: spark thrift server RDD ID overflow lead sql execute failed Key: SPARK-37821 URL: https://issues.apache.org/jira/browse/SPARK-37821 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.2.0 Reporter: muhong this problem will happen in long run spark application，such as thrift server； as only one SparkContext instance in thrift server driver size，so if the concurrency of sql request is large or the sql is too complicate（this will create a lot of rdd）, the rdd will be generate too fast , the rdd id （SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala] ）will be consume fast, after a few months the nextRddId will overflow。the newRddId will be negative number，but the rdd's block id need to be positive, so this will lead a exception"Failed to parse rdd_-2123452330_2 into block ID"，so can not exchange data during sql execution, and lead sql execute failed -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37820) Replace ApacheCommonBase64 with JavaBase64 for fucntions
[ https://issues.apache.org/jira/browse/SPARK-37820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-37820: - Summary: Replace ApacheCommonBase64 with JavaBase64 for fucntions (was: Replace ApacheCommonBase64 with JavaBase64 for ) > Replace ApacheCommonBase64 with JavaBase64 for fucntions > > > Key: SPARK-37820 > URL: https://issues.apache.org/jira/browse/SPARK-37820 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Replace dependency on third-party libraries with native > support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html > ) for Base-64 encode/decode. > 1. Performace gain > http://java-performance.info/base64-encoding-and-decoding-performance/ > 2. reduce dependencies afterward > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37820) Replace ApacheCommonBase64 with JavaBase64 for string fucntions
[ https://issues.apache.org/jira/browse/SPARK-37820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-37820: - Summary: Replace ApacheCommonBase64 with JavaBase64 for string fucntions (was: Replace ApacheCommonBase64 with JavaBase64 for fucntions) > Replace ApacheCommonBase64 with JavaBase64 for string fucntions > --- > > Key: SPARK-37820 > URL: https://issues.apache.org/jira/browse/SPARK-37820 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.0 >Reporter: Kent Yao >Priority: Major > > Replace dependency on third-party libraries with native > support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html > ) for Base-64 encode/decode. > 1. Performace gain > http://java-performance.info/base64-encoding-and-decoding-performance/ > 2. reduce dependencies afterward > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37820) Replace ApacheCommonBase64 with JavaBase64 for
Kent Yao created SPARK-37820: Summary: Replace ApacheCommonBase64 with JavaBase64 for Key: SPARK-37820 URL: https://issues.apache.org/jira/browse/SPARK-37820 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.0 Reporter: Kent Yao Replace dependency on third-party libraries with native support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html ) for Base-64 encode/decode. 1. Performace gain http://java-performance.info/base64-encoding-and-decoding-performance/ 2. reduce dependencies afterward -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37367) Reenable exception test in DDLParserSuite.create view -- basic
[ https://issues.apache.org/jira/browse/SPARK-37367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37367: Assignee: angerszhu > Reenable exception test in DDLParserSuite.create view -- basic > -- > > Key: SPARK-37367 > URL: https://issues.apache.org/jira/browse/SPARK-37367 > Project: Spark > Issue Type: Test > Components: SQL, Tests >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Assignee: angerszhu >Priority: Minor > > SPARK-37308 disabled a test due to unknown flakiness. We should enable this > back after investigation. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37367) Reenable exception test in DDLParserSuite.create view -- basic
[ https://issues.apache.org/jira/browse/SPARK-37367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37367. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35040 [https://github.com/apache/spark/pull/35040] > Reenable exception test in DDLParserSuite.create view -- basic > -- > > Key: SPARK-37367 > URL: https://issues.apache.org/jira/browse/SPARK-37367 > Project: Spark > Issue Type: Test > Components: SQL, Tests >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Assignee: angerszhu >Priority: Minor > Fix For: 3.3.0 > > > SPARK-37308 disabled a test due to unknown flakiness. We should enable this > back after investigation. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37491) Fix Series.asof when values of the series is not sorted
[ https://issues.apache.org/jira/browse/SPARK-37491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469589#comment-17469589 ] Haejoon Lee commented on SPARK-37491: - [~pralabhkumar] Sure, please go ahead! > Fix Series.asof when values of the series is not sorted > --- > > Key: SPARK-37491 > URL: https://issues.apache.org/jira/browse/SPARK-37491 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > > https://github.com/apache/spark/pull/34737#discussion_r758223279 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37812) When deserializing an Orc struct, reuse the result row when possible
[ https://issues.apache.org/jira/browse/SPARK-37812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37812. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35090 [https://github.com/apache/spark/pull/35090] > When deserializing an Orc struct, reuse the result row when possible > > > Key: SPARK-37812 > URL: https://issues.apache.org/jira/browse/SPARK-37812 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Bruce Robbins >Assignee: Bruce Robbins >Priority: Major > Fix For: 3.3.0 > > > For each Orc struct, the Orc deserializer creates a new result row object. > There are cases where the Orc deserializer can reuse the result row object, > resulting in performance gains. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37812) When deserializing an Orc struct, reuse the result row when possible
[ https://issues.apache.org/jira/browse/SPARK-37812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37812: - Assignee: Bruce Robbins > When deserializing an Orc struct, reuse the result row when possible > > > Key: SPARK-37812 > URL: https://issues.apache.org/jira/browse/SPARK-37812 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Bruce Robbins >Assignee: Bruce Robbins >Priority: Major > > For each Orc struct, the Orc deserializer creates a new result row object. > There are cases where the Orc deserializer can reuse the result row object, > resulting in performance gains. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
[ https://issues.apache.org/jira/browse/SPARK-37819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37819: - Assignee: Dongjoon Hyun > Add OUTLIER executor roll policy and use it by default > -- > > Key: SPARK-37819 > URL: https://issues.apache.org/jira/browse/SPARK-37819 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
[ https://issues.apache.org/jira/browse/SPARK-37819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469565#comment-17469565 ] Apache Spark commented on SPARK-37819: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35109 > Add OUTLIER executor roll policy and use it by default > -- > > Key: SPARK-37819 > URL: https://issues.apache.org/jira/browse/SPARK-37819 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
[ https://issues.apache.org/jira/browse/SPARK-37819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37819: Assignee: (was: Apache Spark) > Add OUTLIER executor roll policy and use it by default > -- > > Key: SPARK-37819 > URL: https://issues.apache.org/jira/browse/SPARK-37819 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
[ https://issues.apache.org/jira/browse/SPARK-37819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469564#comment-17469564 ] Apache Spark commented on SPARK-37819: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35109 > Add OUTLIER executor roll policy and use it by default > -- > > Key: SPARK-37819 > URL: https://issues.apache.org/jira/browse/SPARK-37819 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
[ https://issues.apache.org/jira/browse/SPARK-37819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37819: Assignee: Apache Spark > Add OUTLIER executor roll policy and use it by default > -- > > Key: SPARK-37819 > URL: https://issues.apache.org/jira/browse/SPARK-37819 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37819) Add OUTLIER executor roll policy and use it by default
Dongjoon Hyun created SPARK-37819: - Summary: Add OUTLIER executor roll policy and use it by default Key: SPARK-37819 URL: https://issues.apache.org/jira/browse/SPARK-37819 Project: Spark Issue Type: Sub-task Components: Kubernetes Affects Versions: 3.3.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37710) Add detailed error message for java.io.IOException occurring on Kryo flow
[
https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eren Avsarogullari updated SPARK-37710:
---
Description:
*Input/output error* usually points environmental issues such as disk
read/write failures due to disk corruption, network access failures etc. This
PR aims to be added detailed error message to catch this kind of environmental
cases occurring on problematic BlockManager and logs with *BlockManager
hostname, blockId and blockPath* details.
Following stack-trace occurred on disk corruption:
{code:java}
com.esotericsoftware.kryo.KryoException: java.io.IOException: Input/output error
Serialization trace:
buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch)
at com.esotericsoftware.kryo.io.Input.fill(Input.java:166)
at com.esotericsoftware.kryo.io.Input.require(Input.java:196)
at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:346)
at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:326)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:55)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:38)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:381)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:816)
at
org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:296)
at
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:168)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
at
org.apache.spark.storage.BlockManager.maybeCacheDiskValuesInMemory(BlockManager.scala:1569)
at
org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:877)
at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1163)
...
Caused by: java.io.IOException: Input/output error
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at
net.jpountz.lz4.LZ4BlockInputStream.tryReadFully(LZ4BlockInputStream.java:269)
at
net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:280)
at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:243)
at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:157)
at com.esotericsoftware.kryo.io.Input.fill(Input.java:164)
... 87 more {code}
*Proposed Error Message:*
{code:java}
java.io.IOException: Input/output error. BlockManagerId(driver, localhost,
49455, None) - blockId: test_my-block-id - blockDiskPath:
/private/var/folders/kj/mccyycwn6mjdwnglw9g3k6pmgq/T/blockmgr-12dba181-771e-4ff9-a2bc-fa3ce6dbabfa/11/test_my-block-id
{code}
was:
*Input/output error* usually points environmental issues such as disk
read/write failures due to disk corruption, network access failures etc. This
PR aims to be added clear message to catch this kind of environmental cases
occurring on BlockManager and logs with {*}BlockManager hostname, blockId and
blockPath{*}.
Following stack-trace occurred on disk corruption:
{code:java}
com.esotericsoftware.kryo.KryoException: java.io.IOException: Input/output error
Serialization trace:
buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch)
at com.esotericsoftware.kryo.io.Input.fill(Input.java:166)
at com.esotericsoftware.kryo.io.Input.require(Input.java:196)
at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:346)
at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:326)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:55)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:38)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
at
[jira] [Updated] (SPARK-37710) Add detailed error message for java.io.IOException occurring on Kryo flow
[
https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eren Avsarogullari updated SPARK-37710:
---
Summary: Add detailed error message for java.io.IOException occurring on
Kryo flow (was: Add clear error message for java.io.IOException: Input/output
error)
> Add detailed error message for java.io.IOException occurring on Kryo flow
> -
>
> Key: SPARK-37710
> URL: https://issues.apache.org/jira/browse/SPARK-37710
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.1.1
>Reporter: Eren Avsarogullari
>Priority: Major
>
> *Input/output error* usually points environmental issues such as disk
> read/write failures due to disk corruption, network access failures etc. This
> PR aims to be added clear message to catch this kind of environmental cases
> occurring on BlockManager and logs with {*}BlockManager hostname, blockId and
> blockPath{*}.
> Following stack-trace occurred on disk corruption:
> {code:java}
> com.esotericsoftware.kryo.KryoException: java.io.IOException: Input/output
> error
> Serialization trace:
> buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch)
> at com.esotericsoftware.kryo.io.Input.fill(Input.java:166)
> at com.esotericsoftware.kryo.io.Input.require(Input.java:196)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:346)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:326)
> at
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:55)
> at
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:38)
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
> at
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:381)
> at
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:816)
> at
> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:296)
> at
> org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:168)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at
> org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
> at
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
> at
> org.apache.spark.storage.BlockManager.maybeCacheDiskValuesInMemory(BlockManager.scala:1569)
> at
> org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:877)
> at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1163)
> ...
> Caused by: java.io.IOException: Input/output error
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at
> net.jpountz.lz4.LZ4BlockInputStream.tryReadFully(LZ4BlockInputStream.java:269)
> at
> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:280)
> at
> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:243)
> at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:157)
> at com.esotericsoftware.kryo.io.Input.fill(Input.java:164)
> ... 87 more {code}
> *Proposed Error Message:*
> {code:java}
> java.io.IOException: Input/output error usually occurs due to environmental
> problems (e.g: disk corruption, network failure etc) so please check env
> status if healthy. BlockManagerId(driver, localhost, 54937, None) -
> blockName: test_my-block-id - blockDiskPath:
> /private/var/folders/kj/mccyycwn6mjdwnglw9g3k6pmgq/T/blockmgr-e86d8f67-a993-407f-ad3b-3cfb667b4ad4/11/test_my-block-id
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37802) composite field name like `field name` doesn't work with Aggregate push down
[
https://issues.apache.org/jira/browse/SPARK-37802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37802:
Assignee: Apache Spark (was: Huaxin Gao)
> composite field name like `field name` doesn't work with Aggregate push down
>
>
> Key: SPARK-37802
> URL: https://issues.apache.org/jira/browse/SPARK-37802
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Huaxin Gao
>Assignee: Apache Spark
>Priority: Minor
>
> {code:java}
> sql("SELECT SUM(`field name`) FROM h2.test.table")
> org.apache.spark.sql.catalyst.parser.ParseException:
> extraneous input 'name' expecting (line 1, pos 9)
> at
> org.apache.spark.sql.catalyst.parser.ParseErrorListener$.syntaxError(ParseDriver.scala:212)
> at
> org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
> at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.reportUnwantedToken(DefaultErrorStrategy.java:377)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.singleTokenDeletion(DefaultErrorStrategy.java:548)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:467)
> at org.antlr.v4.runtime.Parser.match(Parser.java:206)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser.singleMultipartIdentifier(SqlBaseParser.java:519)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37802) composite field name like `field name` doesn't work with Aggregate push down
[
https://issues.apache.org/jira/browse/SPARK-37802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469508#comment-17469508
]
Apache Spark commented on SPARK-37802:
--
User 'huaxingao' has created a pull request for this issue:
https://github.com/apache/spark/pull/35108
> composite field name like `field name` doesn't work with Aggregate push down
>
>
> Key: SPARK-37802
> URL: https://issues.apache.org/jira/browse/SPARK-37802
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Huaxin Gao
>Assignee: Huaxin Gao
>Priority: Minor
>
> {code:java}
> sql("SELECT SUM(`field name`) FROM h2.test.table")
> org.apache.spark.sql.catalyst.parser.ParseException:
> extraneous input 'name' expecting (line 1, pos 9)
> at
> org.apache.spark.sql.catalyst.parser.ParseErrorListener$.syntaxError(ParseDriver.scala:212)
> at
> org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
> at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.reportUnwantedToken(DefaultErrorStrategy.java:377)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.singleTokenDeletion(DefaultErrorStrategy.java:548)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:467)
> at org.antlr.v4.runtime.Parser.match(Parser.java:206)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser.singleMultipartIdentifier(SqlBaseParser.java:519)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37802) composite field name like `field name` doesn't work with Aggregate push down
[
https://issues.apache.org/jira/browse/SPARK-37802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37802:
Assignee: Huaxin Gao (was: Apache Spark)
> composite field name like `field name` doesn't work with Aggregate push down
>
>
> Key: SPARK-37802
> URL: https://issues.apache.org/jira/browse/SPARK-37802
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Huaxin Gao
>Assignee: Huaxin Gao
>Priority: Minor
>
> {code:java}
> sql("SELECT SUM(`field name`) FROM h2.test.table")
> org.apache.spark.sql.catalyst.parser.ParseException:
> extraneous input 'name' expecting (line 1, pos 9)
> at
> org.apache.spark.sql.catalyst.parser.ParseErrorListener$.syntaxError(ParseDriver.scala:212)
> at
> org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
> at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.reportUnwantedToken(DefaultErrorStrategy.java:377)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.singleTokenDeletion(DefaultErrorStrategy.java:548)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:467)
> at org.antlr.v4.runtime.Parser.match(Parser.java:206)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser.singleMultipartIdentifier(SqlBaseParser.java:519)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37802) composite field name like `field name` doesn't work with Aggregate push down
[
https://issues.apache.org/jira/browse/SPARK-37802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469509#comment-17469509
]
Apache Spark commented on SPARK-37802:
--
User 'huaxingao' has created a pull request for this issue:
https://github.com/apache/spark/pull/35108
> composite field name like `field name` doesn't work with Aggregate push down
>
>
> Key: SPARK-37802
> URL: https://issues.apache.org/jira/browse/SPARK-37802
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Huaxin Gao
>Assignee: Huaxin Gao
>Priority: Minor
>
> {code:java}
> sql("SELECT SUM(`field name`) FROM h2.test.table")
> org.apache.spark.sql.catalyst.parser.ParseException:
> extraneous input 'name' expecting (line 1, pos 9)
> at
> org.apache.spark.sql.catalyst.parser.ParseErrorListener$.syntaxError(ParseDriver.scala:212)
> at
> org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
> at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.reportUnwantedToken(DefaultErrorStrategy.java:377)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.singleTokenDeletion(DefaultErrorStrategy.java:548)
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:467)
> at org.antlr.v4.runtime.Parser.match(Parser.java:206)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser.singleMultipartIdentifier(SqlBaseParser.java:519)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37788) ColumnOrName vs Column in PySpark Functions module
[
https://issues.apache.org/jira/browse/SPARK-37788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maciej Szymkiewicz resolved SPARK-37788.
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 35071
[https://github.com/apache/spark/pull/35071]
> ColumnOrName vs Column in PySpark Functions module
> --
>
> Key: SPARK-37788
> URL: https://issues.apache.org/jira/browse/SPARK-37788
> Project: Spark
> Issue Type: Question
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Daniel Davies
>Assignee: Daniel Davies
>Priority: Minor
> Fix For: 3.3.0
>
>
> PySpark has mainly migrated to supporting both Column input types as well as
> string names of columns ("ColumnOrName") in it's functions module. There seem
> to be a small number of functions that need updating; either on conversions
> of input string names representing columns into the Column type, or simple
> annotation changes that indicate the function supports column string names.
> Below are the functions I've seen:
> * F.overlay: Annotation only
> * F.least: Annotation only
> * F.slice: Needs a conversion
> * F.array_repeat: Needs a conversion
> See here for additional context:
> [https://github.com/apache/spark/pull/35032#issuecomment-1003033776]
> I'm happy to make a quick PR fixing these, if there is no reason for these
> functions being handled as a special case.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37788) ColumnOrName vs Column in PySpark Functions module
[
https://issues.apache.org/jira/browse/SPARK-37788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maciej Szymkiewicz reassigned SPARK-37788:
--
Assignee: Daniel Davies
> ColumnOrName vs Column in PySpark Functions module
> --
>
> Key: SPARK-37788
> URL: https://issues.apache.org/jira/browse/SPARK-37788
> Project: Spark
> Issue Type: Question
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Daniel Davies
>Assignee: Daniel Davies
>Priority: Minor
>
> PySpark has mainly migrated to supporting both Column input types as well as
> string names of columns ("ColumnOrName") in it's functions module. There seem
> to be a small number of functions that need updating; either on conversions
> of input string names representing columns into the Column type, or simple
> annotation changes that indicate the function supports column string names.
> Below are the functions I've seen:
> * F.overlay: Annotation only
> * F.least: Annotation only
> * F.slice: Needs a conversion
> * F.array_repeat: Needs a conversion
> See here for additional context:
> [https://github.com/apache/spark/pull/35032#issuecomment-1003033776]
> I'm happy to make a quick PR fixing these, if there is no reason for these
> functions being handled as a special case.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37491) Fix Series.asof when values of the series is not sorted
[ https://issues.apache.org/jira/browse/SPARK-37491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469471#comment-17469471 ] pralabhkumar commented on SPARK-37491: -- I would like to work on this . Basically the problem is in series.py , finding Max . cond = [ F.max(F.when(index_scol <= SF.lit(index).cast(index_type), self.spark.column)) for index in where ] cc [~hyukjin.kwon] [~itholic] > Fix Series.asof when values of the series is not sorted > --- > > Key: SPARK-37491 > URL: https://issues.apache.org/jira/browse/SPARK-37491 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > > https://github.com/apache/spark/pull/34737#discussion_r758223279 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-37491) Fix Series.asof when values of the series is not sorted
[ https://issues.apache.org/jira/browse/SPARK-37491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469471#comment-17469471 ] pralabhkumar edited comment on SPARK-37491 at 1/5/22, 6:27 PM: --- I would like to work on this . Basically the problem is in series.py . We should not find max here. cond = [ F.max(F.when(index_scol <= SF.lit(index).cast(index_type), self.spark.column)) for index in where ] cc [~hyukjin.kwon] [~itholic] was (Author: pralabhkumar): I would like to work on this . Basically the problem is in series.py , finding Max . cond = [ F.max(F.when(index_scol <= SF.lit(index).cast(index_type), self.spark.column)) for index in where ] cc [~hyukjin.kwon] [~itholic] > Fix Series.asof when values of the series is not sorted > --- > > Key: SPARK-37491 > URL: https://issues.apache.org/jira/browse/SPARK-37491 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > > https://github.com/apache/spark/pull/34737#discussion_r758223279 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36879) Support Parquet v2 data page encodings for the vectorized path
[
https://issues.apache.org/jira/browse/SPARK-36879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun reassigned SPARK-36879:
Assignee: Parth Chandra
> Support Parquet v2 data page encodings for the vectorized path
> --
>
> Key: SPARK-36879
> URL: https://issues.apache.org/jira/browse/SPARK-36879
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Chao Sun
>Assignee: Parth Chandra
>Priority: Major
> Fix For: 3.3.0
>
>
> Currently Spark only support Parquet V1 encodings (i.e.,
> PLAIN/DICTIONARY/RLE) in the vectorized path, and throws exception otherwise:
> {code}
> java.lang.UnsupportedOperationException: Unsupported encoding:
> DELTA_BYTE_ARRAY
> {code}
> It will be good to support v2 encodings too, including DELTA_BINARY_PACKED,
> DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY as well as BYTE_STREAM_SPLIT as
> listed in https://github.com/apache/parquet-format/blob/master/Encodings.md
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36879) Support Parquet v2 data page encodings for the vectorized path
[
https://issues.apache.org/jira/browse/SPARK-36879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved SPARK-36879.
--
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 34471
[https://github.com/apache/spark/pull/34471]
> Support Parquet v2 data page encodings for the vectorized path
> --
>
> Key: SPARK-36879
> URL: https://issues.apache.org/jira/browse/SPARK-36879
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Chao Sun
>Priority: Major
> Fix For: 3.3.0
>
>
> Currently Spark only support Parquet V1 encodings (i.e.,
> PLAIN/DICTIONARY/RLE) in the vectorized path, and throws exception otherwise:
> {code}
> java.lang.UnsupportedOperationException: Unsupported encoding:
> DELTA_BYTE_ARRAY
> {code}
> It will be good to support v2 encodings too, including DELTA_BINARY_PACKED,
> DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY as well as BYTE_STREAM_SPLIT as
> listed in https://github.com/apache/parquet-format/blob/master/Encodings.md
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform
[ https://issues.apache.org/jira/browse/SPARK-37796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37796: - Priority: Minor (was: Major) > ByteArrayMethods arrayEquals should fast skip the check of aligning with > unaligned platform > --- > > Key: SPARK-37796 > URL: https://issues.apache.org/jira/browse/SPARK-37796 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: XiDuo You >Assignee: XiDuo You >Priority: Minor > Fix For: 3.3.0 > > > The method `arrayEquals` in `ByteArrayMethods` is critical function which is > used in `UTF8String.` `equals`, `indexOf`,`find` etc. > After SPARK-16962, it add the complexity of aligned. It would be better to > fast sikip the check of aligning if the platform is unaligned. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform
[ https://issues.apache.org/jira/browse/SPARK-37796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37796: Assignee: XiDuo You > ByteArrayMethods arrayEquals should fast skip the check of aligning with > unaligned platform > --- > > Key: SPARK-37796 > URL: https://issues.apache.org/jira/browse/SPARK-37796 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: XiDuo You >Assignee: XiDuo You >Priority: Major > Fix For: 3.3.0 > > > The method `arrayEquals` in `ByteArrayMethods` is critical function which is > used in `UTF8String.` `equals`, `indexOf`,`find` etc. > After SPARK-16962, it add the complexity of aligned. It would be better to > fast sikip the check of aligning if the platform is unaligned. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform
[ https://issues.apache.org/jira/browse/SPARK-37796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37796. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35078 [https://github.com/apache/spark/pull/35078] > ByteArrayMethods arrayEquals should fast skip the check of aligning with > unaligned platform > --- > > Key: SPARK-37796 > URL: https://issues.apache.org/jira/browse/SPARK-37796 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: XiDuo You >Priority: Major > Fix For: 3.3.0 > > > The method `arrayEquals` in `ByteArrayMethods` is critical function which is > used in `UTF8String.` `equals`, `indexOf`,`find` etc. > After SPARK-16962, it add the complexity of aligned. It would be better to > fast sikip the check of aligning if the platform is unaligned. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37817) Remove unreachable code in complexTypeExtractors.scala
[ https://issues.apache.org/jira/browse/SPARK-37817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-37817. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35106 [https://github.com/apache/spark/pull/35106] > Remove unreachable code in complexTypeExtractors.scala > --- > > Key: SPARK-37817 > URL: https://issues.apache.org/jira/browse/SPARK-37817 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Trivial > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37818) Add option for show create table command
[ https://issues.apache.org/jira/browse/SPARK-37818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469245#comment-17469245 ] Apache Spark commented on SPARK-37818: -- User 'Peng-Lei' has created a pull request for this issue: https://github.com/apache/spark/pull/35107 > Add option for show create table command > > > Key: SPARK-37818 > URL: https://issues.apache.org/jira/browse/SPARK-37818 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.3.0 >Reporter: PengLei >Priority: Trivial > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37818) Add option for show create table command
[ https://issues.apache.org/jira/browse/SPARK-37818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469244#comment-17469244 ] Apache Spark commented on SPARK-37818: -- User 'Peng-Lei' has created a pull request for this issue: https://github.com/apache/spark/pull/35107 > Add option for show create table command > > > Key: SPARK-37818 > URL: https://issues.apache.org/jira/browse/SPARK-37818 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.3.0 >Reporter: PengLei >Priority: Trivial > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37818) Add option for show create table command
[ https://issues.apache.org/jira/browse/SPARK-37818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37818: Assignee: Apache Spark > Add option for show create table command > > > Key: SPARK-37818 > URL: https://issues.apache.org/jira/browse/SPARK-37818 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.3.0 >Reporter: PengLei >Assignee: Apache Spark >Priority: Trivial > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37818) Add option for show create table command
[ https://issues.apache.org/jira/browse/SPARK-37818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37818: Assignee: (was: Apache Spark) > Add option for show create table command > > > Key: SPARK-37818 > URL: https://issues.apache.org/jira/browse/SPARK-37818 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.3.0 >Reporter: PengLei >Priority: Trivial > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37818) Add option for show create table command
PengLei created SPARK-37818: --- Summary: Add option for show create table command Key: SPARK-37818 URL: https://issues.apache.org/jira/browse/SPARK-37818 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 3.3.0 Reporter: PengLei Fix For: 3.3.0 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36593) [Deprecated] Support the Volcano Job API
[ https://issues.apache.org/jira/browse/SPARK-36593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36593: Summary: [Deprecated] Support the Volcano Job API (was: Support the Volcano Job API) > [Deprecated] Support the Volcano Job API > > > Key: SPARK-36593 > URL: https://issues.apache.org/jira/browse/SPARK-36593 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Holden Karau >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36061) Add Volcano feature step
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36061: Description: Create a PodGroup with user specified minimum resources required > Add Volcano feature step > > > Key: SPARK-36061 > URL: https://issues.apache.org/jira/browse/SPARK-36061 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Holden Karau >Priority: Major > > Create a PodGroup with user specified minimum resources required -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36061) Add Volcano feature step
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36061: Summary: Add Volcano feature step (was: Create a PodGroup with user specified minimum resources required) > Add Volcano feature step > > > Key: SPARK-36061 > URL: https://issues.apache.org/jira/browse/SPARK-36061 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Holden Karau >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469178#comment-17469178 ] Steve Loughran commented on SPARK-37814: be good to link to all issues related to this, e.g test log problems, so other projects doing the same work know what to look for > Migrating from log4j 1 to log4j 2 > - > > Key: SPARK-37814 > URL: https://issues.apache.org/jira/browse/SPARK-37814 > Project: Spark > Issue Type: Umbrella > Components: Build >Affects Versions: 3.3.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: releasenotes > Fix For: 3.3.0 > > > This is umbrella ticket for all tasks related to migrating to log4j2. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37816) Create table like should change to current user
[ https://issues.apache.org/jira/browse/SPARK-37816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu resolved SPARK-37816. --- Resolution: Won't Fix > Create table like should change to current user > --- > > Key: SPARK-37816 > URL: https://issues.apache.org/jira/browse/SPARK-37816 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: angerszhu >Priority: Major > > Create table like should change to current user. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37817) Remove unreachable code in complexTypeExtractors.scala
[ https://issues.apache.org/jira/browse/SPARK-37817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469159#comment-17469159 ] Apache Spark commented on SPARK-37817: -- User 'gengliangwang' has created a pull request for this issue: https://github.com/apache/spark/pull/35106 > Remove unreachable code in complexTypeExtractors.scala > --- > > Key: SPARK-37817 > URL: https://issues.apache.org/jira/browse/SPARK-37817 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37817) Remove unreachable code in complexTypeExtractors.scala
[ https://issues.apache.org/jira/browse/SPARK-37817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37817: Assignee: Gengliang Wang (was: Apache Spark) > Remove unreachable code in complexTypeExtractors.scala > --- > > Key: SPARK-37817 > URL: https://issues.apache.org/jira/browse/SPARK-37817 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37817) Remove unreachable code in complexTypeExtractors.scala
[ https://issues.apache.org/jira/browse/SPARK-37817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469157#comment-17469157 ] Apache Spark commented on SPARK-37817: -- User 'gengliangwang' has created a pull request for this issue: https://github.com/apache/spark/pull/35106 > Remove unreachable code in complexTypeExtractors.scala > --- > > Key: SPARK-37817 > URL: https://issues.apache.org/jira/browse/SPARK-37817 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37817) Remove unreachable code in complexTypeExtractors.scala
[ https://issues.apache.org/jira/browse/SPARK-37817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37817: Assignee: Apache Spark (was: Gengliang Wang) > Remove unreachable code in complexTypeExtractors.scala > --- > > Key: SPARK-37817 > URL: https://issues.apache.org/jira/browse/SPARK-37817 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Assignee: Apache Spark >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37817) Remove unreachable code in complexTypeExtractors.scala
Gengliang Wang created SPARK-37817: -- Summary: Remove unreachable code in complexTypeExtractors.scala Key: SPARK-37817 URL: https://issues.apache.org/jira/browse/SPARK-37817 Project: Spark Issue Type: Task Components: SQL Affects Versions: 3.3.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37816) Create table like should change to current user
angerszhu created SPARK-37816: - Summary: Create table like should change to current user Key: SPARK-37816 URL: https://issues.apache.org/jira/browse/SPARK-37816 Project: Spark Issue Type: Task Components: SQL Affects Versions: 3.2.0, 3.1.2 Reporter: angerszhu Create table like should change to current user. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37815) Fix the github action job "test_report"
[ https://issues.apache.org/jira/browse/SPARK-37815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-37815. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35104 [https://github.com/apache/spark/pull/35104] > Fix the github action job "test_report" > --- > > Key: SPARK-37815 > URL: https://issues.apache.org/jira/browse/SPARK-37815 > Project: Spark > Issue Type: Task > Components: Project Infra >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Priority: Minor > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37815) Fix the github action job "test_report"
[ https://issues.apache.org/jira/browse/SPARK-37815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-37815: -- Assignee: Gengliang Wang > Fix the github action job "test_report" > --- > > Key: SPARK-37815 > URL: https://issues.apache.org/jira/browse/SPARK-37815 > Project: Spark > Issue Type: Task > Components: Project Infra >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37800) TreeNode.argString incorrectly formats arguments of type Set[_]
[
https://issues.apache.org/jira/browse/SPARK-37800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469141#comment-17469141
]
Apache Spark commented on SPARK-37800:
--
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/35105
> TreeNode.argString incorrectly formats arguments of type Set[_]
> ---
>
> Key: SPARK-37800
> URL: https://issues.apache.org/jira/browse/SPARK-37800
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Simeon Simeonov
>Assignee: Simeon Simeonov
>Priority: Minor
> Fix For: 3.2.1, 3.3.0
>
>
> The implementation of {{argString}} uses the following pattern for sets:
>
> {code:java}
> case set: Set[_] =>
> // Sort elements for deterministic behaviours
> val sortedSeq = set.toSeq.map(formatArg(_, maxFields).sorted)
>
> truncatedString(sortedSeq, "{", ", ", "}", maxFields) :: Nil {code}
> Instead of sorting the elements of the set, the implementation sorts the
> characters of the strings that {{formatArg}} returns.
> The fix is simply to move the closing parenthesis to the correct location:
> {code:java}
> val sortedSeq = set.toSeq.map(formatArg(_, maxFields)).sorted
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37779) Make ColumnarToRowExec plan canonicalizable after (de)serialization
[
https://issues.apache.org/jira/browse/SPARK-37779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-37779:
-
Fix Version/s: (was: 3.1.3)
(was: 3.0.4)
> Make ColumnarToRowExec plan canonicalizable after (de)serialization
> ---
>
> Key: SPARK-37779
> URL: https://issues.apache.org/jira/browse/SPARK-37779
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Assignee: Hyukjin Kwon
>Priority: Minor
> Fix For: 3.2.1, 3.3.0
>
>
> SPARK-23731 fixed the plans to be serializable by leveraging lazy but
> SPARK-28213 introduced new code path that calls the lazy val which triggers
> null point exception in
> https://github.com/apache/spark/blob/77b164aac9764049a4820064421ef82ec0bc14fb/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala#L68
> This can fail during canonicalization during, for example, eliminating sub
> common expressions:
> {code}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.execution.FileSourceScanExec.supportsColumnar$lzycompute(DataSourceScanExec.scala:280)
> at
> org.apache.spark.sql.execution.FileSourceScanExec.supportsColumnar(DataSourceScanExec.scala:279)
> at
> org.apache.spark.sql.execution.InputAdapter.supportsColumnar(WholeStageCodegenExec.scala:509)
> at
> org.apache.spark.sql.execution.ColumnarToRowExec.(Columnar.scala:67)
> ...
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.canonicalized$lzycompute(QueryPlan.scala:581)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.canonicalized(QueryPlan.scala:580)
> at
> org.apache.spark.sql.execution.ScalarSubquery.canonicalized$lzycompute(subquery.scala:110)
> ...
> at
> org.apache.spark.sql.catalyst.expressions.ExpressionEquals.hashCode(EquivalentExpressions.scala:275)
> ...
> at scala.collection.mutable.HashTable.findEntry$(HashTable.scala:135)
> at scala.collection.mutable.HashMap.findEntry(HashMap.scala:44)
> at scala.collection.mutable.HashMap.get(HashMap.scala:74)
> at
> org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.addExpr(EquivalentExpressions.scala:46)
> at
> org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.addExprTreeHelper$1(EquivalentExpressions.scala:147)
> at
> org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.addExprTree(EquivalentExpressions.scala:170)
> at
> org.apache.spark.sql.catalyst.expressions.SubExprEvaluationRuntime.$anonfun$proxyExpressions$1(SubExprEvaluationRuntime.scala:89)
> at
> org.apache.spark.sql.catalyst.expressions.SubExprEvaluationRuntime.$anonfun$proxyExpressions$1$adapted(SubExprEvaluationRuntime.scala:89)
> at scala.collection.immutable.List.foreach(List.scala:392)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37815) Fix the github action job "test_report"
[ https://issues.apache.org/jira/browse/SPARK-37815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469114#comment-17469114 ] Apache Spark commented on SPARK-37815: -- User 'gengliangwang' has created a pull request for this issue: https://github.com/apache/spark/pull/35104 > Fix the github action job "test_report" > --- > > Key: SPARK-37815 > URL: https://issues.apache.org/jira/browse/SPARK-37815 > Project: Spark > Issue Type: Task > Components: Project Infra >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37815) Fix the github action job "test_report"
[ https://issues.apache.org/jira/browse/SPARK-37815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469113#comment-17469113 ] Apache Spark commented on SPARK-37815: -- User 'gengliangwang' has created a pull request for this issue: https://github.com/apache/spark/pull/35104 > Fix the github action job "test_report" > --- > > Key: SPARK-37815 > URL: https://issues.apache.org/jira/browse/SPARK-37815 > Project: Spark > Issue Type: Task > Components: Project Infra >Affects Versions: 3.3.0 >Reporter: Gengliang Wang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
