zozo123 commented on code in PR #59231:
URL: https://github.com/apache/spark/pull/59231#discussion_r4221124373
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/externalUDF/PlanExternalUDFsSuite.scala:
##########
@@ -716,25 +719,47 @@ class PlanExternalUDFsSuite extends QueryTest with
SharedSparkSession {
assert(execution.workerSpec == spec)
}
- test("scalar external UDF execution reaches dispatcher creation") {
- val spec = workerSpec("unimplemented-scalar")
- val range = Range(0, 10, 1, 1)
- val function = udf("unimplemented-scalar", spec, Seq(singleOutput(range)))
- val resultAttr = AttributeReference("externalUDF", IntegerType, nullable =
true)()
+ test("sql tests expose one unshaded epoll native library") {
+ val resource = "META-INF/native/libnetty_transport_native_epoll_x86_64.so"
+ val urls = new java.util.ArrayList[java.net.URL]()
+ val found = getClass.getClassLoader.getResources(resource)
+ while (found.hasMoreElements) {
+ urls.add(found.nextElement())
+ }
+ assert(urls.size == 1, urls.toString)
+ assert(urls.get(0).toString.contains("netty-transport-native-epoll"))
+ }
+
+ test("scalar external UDF returns rows from a direct gRPC worker process") {
Review Comment:
Addressed in 26e55e57. `PlanExternalUDFsSuite` assumes on
`EchoGrpcWorkerMain.udsTransportAvailable`, which calls
`UnixDomainSocketTransport.detect()` and returns false on
`UnsupportedOperationException`, matching
`DirectGrpcDispatcherIntegrationSuite`. On this Mac the end-to-end test ran and
checked the identity rows. I do not have a host where `detect()` throws, so the
cancel path is that same exception branch.
##########
sql/core/pom.xml:
##########
@@ -390,6 +403,92 @@
</execution>
</executions>
</plugin>
+ <!--
+ snowflake-jdbc 4.4.0 shades Woodstox and registers it as a StAX
factory.
+ ScalaTest's forked JVM uses Maven's test classpath, so XMLInputFactory
+ then parses sql XML tests with Woodstox. Nested arrays come back with
+ _corrupt_record, and numeric inference changes. Point the test artifact
+ at a jar with those factory registrations removed. Shaded Woodstox
+ classes stay for Snowflake.
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-snowflake-jdbc-for-stax-strip</id>
+ <phase>generate-test-resources</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <artifactItem>
+ <groupId>net.snowflake</groupId>
+ <artifactId>snowflake-jdbc</artifactId>
+ <version>${snowflake.jdbc.version}</version>
+
<outputDirectory>${project.build.directory}/test-classpath</outputDirectory>
+ <destFileName>snowflake-jdbc-original.jar</destFileName>
Review Comment:
Addressed in 26e55e57. The copy target, the zip source, the zip destination,
and the Groovy swap all include `snowflake.jdbc.version`
(`snowflake-jdbc-<version>-original.jar` and
`snowflake-jdbc-<version>-no-stax-services.jar`). A version switch without `mvn
clean` uses a different pair of files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
