[
https://issues.apache.org/jira/browse/SPARK-45029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-45029:
-----------------------------
Description:
The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module
when testing the connect module. This allows `mvn clean install` and `mvn
package test` to successfully pass tests. But if `mvn clean test` is executed
directly, an error in `package org.sparkproject.spark_protobuf.protobuf does
not exist` will occur. This is because `mvn clean test` directly uses the
classes file of the `spark-protobuf` module for testing, without the 'package',
hence it does not `shade` and `relocate` protobuf.
On the other hand, SPARK-43646 breaks the usability of importing Spark as a
Maven project into IDEA.
Therefore, the changes of SPARK-43646 be reverted in
[https://github.com/apache/spark/pull/42746].
In summary, when using Maven to test the connect module:
* sometimes the shaded spark-protobuf jar will be used, such as 'mvn package
test',
* sometimes the unshaded classes directory will be used, such as 'mvn clean
test'.
Therefore, there is no perfect solution to pass both tests, so 'from_protobuf
messageClassName' and 'from_protobuf messageClassName' in
'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when
we find a better solution.
was:
The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module
when testing the connect module. This allows `mvn clean install` and `mvn
package test` to successfully pass tests. But if `mvn clean test` is executed
directly, an error in `package org.sparkproject.spark_protobuf.protobuf does
not exist` will occur. This is because `mvn clean test` directly uses the
classes file of the `spark-protobuf` module for testing, without the 'package',
hence it does not `shade` and `relocate` protobuf.
On the other hand, SPARK-43646 breaks the usability of importing Spark as a
Maven project into IDEA.
Therefore, we revert the changes of SPARK-43646 in
[https://github.com/apache/spark/pull/42746].
In summary, when using Maven to test the connect module:
* sometimes the shaded spark-protobuf jar will be used, such as 'mvn package
test',
* sometimes the unshaded classes directory will be used, such as 'mvn clean
test'.
Therefore, there is no perfect solution to pass both tests, so 'from_protobuf
messageClassName' and 'from_protobuf messageClassName' in
'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when
we find a better solution.
> Ignore `from_protobuf messageClassName` and `from_protobuf messageClassName
> options` in `PlanGenerationTestSuite`
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-45029
> URL: https://issues.apache.org/jira/browse/SPARK-45029
> Project: Spark
> Issue Type: Improvement
> Components: Connect, Tests
> Affects Versions: 3.5.0, 4.0.0
> Reporter: Yang Jie
> Priority: Major
>
> The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf`
> module when testing the connect module. This allows `mvn clean install` and
> `mvn package test` to successfully pass tests. But if `mvn clean test` is
> executed directly, an error in `package
> org.sparkproject.spark_protobuf.protobuf does not exist` will occur. This is
> because `mvn clean test` directly uses the classes file of the
> `spark-protobuf` module for testing, without the 'package', hence it does not
> `shade` and `relocate` protobuf.
> On the other hand, SPARK-43646 breaks the usability of importing Spark as a
> Maven project into IDEA.
> Therefore, the changes of SPARK-43646 be reverted in
> [https://github.com/apache/spark/pull/42746].
> In summary, when using Maven to test the connect module:
> * sometimes the shaded spark-protobuf jar will be used, such as 'mvn package
> test',
> * sometimes the unshaded classes directory will be used, such as 'mvn clean
> test'.
> Therefore, there is no perfect solution to pass both tests, so 'from_protobuf
> messageClassName' and 'from_protobuf messageClassName' in
> 'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled
> when we find a better solution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
