zozo123 commented on code in PR #59231:
URL: https://github.com/apache/spark/pull/59231#discussion_r4221124373
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/externalUDF/PlanExternalUDFsSuite.scala:
##########
@@ -716,25 +719,47 @@ class PlanExternalUDFsSuite extends QueryTest with
SharedSparkSession {
assert(execution.workerSpec == spec)
}
- test("scalar external UDF execution reaches dispatcher creation") {
- val spec = workerSpec("unimplemented-scalar")
- val range = Range(0, 10, 1, 1)
- val function = udf("unimplemented-scalar", spec, Seq(singleOutput(range)))
- val resultAttr = AttributeReference("externalUDF", IntegerType, nullable =
true)()
+ test("sql tests expose one unshaded epoll native library") {
+ val resource = "META-INF/native/libnetty_transport_native_epoll_x86_64.so"
+ val urls = new java.util.ArrayList[java.net.URL]()
+ val found = getClass.getClassLoader.getResources(resource)
+ while (found.hasMoreElements) {
+ urls.add(found.nextElement())
+ }
+ assert(urls.size == 1, urls.toString)
+ assert(urls.get(0).toString.contains("netty-transport-native-epoll"))
+ }
+
+ test("scalar external UDF returns rows from a direct gRPC worker process") {
Review Comment:
Addressed in `ea1f5d30`. `PlanExternalUDFsSuite` assumes on
`EchoGrpcWorkerMain.udsTransportAvailable`, which calls
`UnixDomainSocketTransport.detect()` and returns false on
`UnsupportedOperationException`, matching
`DirectGrpcDispatcherIntegrationSuite`. On this Mac the end-to-end test ran and
checked the identity rows. I do not have a host where `detect()` throws, so the
cancel path is that same exception branch.
##########
sql/core/pom.xml:
##########
@@ -390,6 +403,92 @@
</execution>
</executions>
</plugin>
+ <!--
+ snowflake-jdbc 4.4.0 shades Woodstox and registers it as a StAX
factory.
+ ScalaTest's forked JVM uses Maven's test classpath, so XMLInputFactory
+ then parses sql XML tests with Woodstox. Nested arrays come back with
+ _corrupt_record, and numeric inference changes. Point the test artifact
+ at a jar with those factory registrations removed. Shaded Woodstox
+ classes stay for Snowflake.
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-snowflake-jdbc-for-stax-strip</id>
+ <phase>generate-test-resources</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <artifactItem>
+ <groupId>net.snowflake</groupId>
+ <artifactId>snowflake-jdbc</artifactId>
+ <version>${snowflake.jdbc.version}</version>
+
<outputDirectory>${project.build.directory}/test-classpath</outputDirectory>
+ <destFileName>snowflake-jdbc-original.jar</destFileName>
Review Comment:
Addressed in `ea1f5d30`. The copy target, the zip source, the zip
destination, and the Groovy swap all include `snowflake.jdbc.version`
(`snowflake-jdbc-<version>-original.jar` and
`snowflake-jdbc-<version>-no-stax-services.jar`). A version switch without `mvn
clean` uses a different pair of files.
##########
sql/core/pom.xml:
##########
@@ -315,6 +315,19 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
Review Comment:
Addressed in `ea1f5d30`, which is also the tip of #59155. `sql` now depends
on `udf_worker` in `dev/sparktestsupport/modules.py`. A change under
`udf/worker/` selects `sql` and the modules that already depend on `sql`. A
profiler-only change still selects only `profiler`, and a `sql/core` change
does not select `udf-worker`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
