This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c5fd509ad3c0 [SPARK-49085][CONNECT][BUILD][FOLLOWUP] Remove the
erroneous `type` definition for `spark-protobuf` from
`sql/connect/server/pom.xml`
c5fd509ad3c0 is described below
commit c5fd509ad3c0781f8511986f3c4e52692bc783c4
Author: yangjie01 <[email protected]>
AuthorDate: Wed Sep 11 10:57:35 2024 -0700
[SPARK-49085][CONNECT][BUILD][FOLLOWUP] Remove the erroneous `type`
definition for `spark-protobuf` from `sql/connect/server/pom.xml`
### What changes were proposed in this pull request?
This pr corrects the erroneous changes made in
https://github.com/apache/spark/pull/48051/files, remove the erroneous `type`
definition for `spark-protobuf` from `sql/connect/server/pom.xml`.
### Why are the changes needed?
When maven testing the connect server module, what we need is
`spark-protobuf_2.13-4.0.0-SNAPSHOT.jar` rather than
`spark-protobuf_2.13-4.0.0-SNAPSHOT-tests.jar`.
And https://github.com/apache/spark/pull/48051/files caused the failure of
the Maven daily test:
- https://github.com/apache/spark/actions/runs/10812252676/job/30002163824
```
- from_protobuf_messageClassName_descFilePath *** FAILED ***
org.apache.spark.sql.AnalysisException:
[PROTOBUF_NOT_LOADED_SQL_FUNCTIONS_UNUSABLE] Cannot call the FROM_PROTOBUF SQL
function because the Protobuf data source is not loaded.
Please restart your job or session with the 'spark-protobuf' package
loaded, such as by using the --packages argument on the command line, and then
retry your query or command again. SQLSTATE: 22KD3
at
org.apache.spark.sql.errors.QueryCompilationErrors$.protobufNotLoadedSqlFunctionsUnusable(QueryCompilationErrors.scala:4096)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.liftedTree1$1(toFromProtobufSqlFunctions.scala:184)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.replacement$lzycompute(toFromProtobufSqlFunctions.scala:178)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.replacement(toFromProtobufSqlFunctions.scala:157)
at
org.apache.spark.sql.catalyst.expressions.RuntimeReplaceable.dataType(Expression.scala:417)
at
org.apache.spark.sql.catalyst.expressions.RuntimeReplaceable.dataType$(Expression.scala:417)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.dataType(toFromProtobufSqlFunctions.scala:86)
at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:195)
at
org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:74)
at scala.collection.immutable.List.map(List.scala:247)
...
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual check:
```
build/mvn clean install -DskipTests -Phive
build/mvn test -pl sql/connect/server
```
**Before**
```
...
- from_protobuf_messageClassName_descFilePath_options *** FAILED ***
org.apache.spark.sql.AnalysisException:
[PROTOBUF_NOT_LOADED_SQL_FUNCTIONS_UNUSABLE] Cannot call the FROM_PROTOBUF SQL
function because the Protobuf data source is not loaded.
Please restart your job or session with the 'spark-protobuf' package
loaded, such as by using the --packages argument on the command line, and then
retry your query or command again. SQLSTATE: 22KD3
at
org.apache.spark.sql.errors.QueryCompilationErrors$.protobufNotLoadedSqlFunctionsUnusable(QueryCompilationErrors.scala:4096)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.liftedTree1$1(toFromProtobufSqlFunctions.scala:184)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.replacement$lzycompute(toFromProtobufSqlFunctions.scala:178)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.replacement(toFromProtobufSqlFunctions.scala:157)
at
org.apache.spark.sql.catalyst.expressions.RuntimeReplaceable.dataType(Expression.scala:417)
at
org.apache.spark.sql.catalyst.expressions.RuntimeReplaceable.dataType$(Expression.scala:417)
at
org.apache.spark.sql.catalyst.expressions.FromProtobuf.dataType(toFromProtobufSqlFunctions.scala:86)
at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:195)
at
org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:74)
at scala.collection.immutable.List.map(List.scala:247)
...
Run completed in 2 minutes, 19 seconds.
Total number of tests run: 948
Suites: completed 27, aborted 0
Tests: succeeded 942, failed 6, canceled 0, ignored 0, pending 0
*** 6 TESTS FAILED ***
```
**After**
```
Run completed in 2 minutes, 2 seconds.
Total number of tests run: 948
Suites: completed 27, aborted 0
Tests: succeeded 948, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #48080 from LuciferYang/SPARK-49085-FOLLOWUP-2.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
sql/connect/server/pom.xml | 1 -
1 file changed, 1 deletion(-)
diff --git a/sql/connect/server/pom.xml b/sql/connect/server/pom.xml
index 110065fcd972..3350c4261e9d 100644
--- a/sql/connect/server/pom.xml
+++ b/sql/connect/server/pom.xml
@@ -111,7 +111,6 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-protobuf_${scala.binary.version}</artifactId>
<version>${project.version}</version>
- <type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
