dillitz opened a new pull request, #54016: URL: https://github.com/apache/spark/pull/54016
### What changes were proposed in this pull request? Automatically use the `Metadata.BINARY_BYTE_MARSHALLER` for `-bin` suffixed header keys, assuming base64-encoded header value strings set through the Scala Spark Connect client builder. ### Why are the changes needed? The Scala Spark Connect client currently only allows setting `Metadata.ASCII_STRING_MARSHALLER` headers and fails if one tries to put a (binary) header with `-bin` key suffix: ``` [info] java.lang.IllegalArgumentException: ASCII header is named test-bin. Only binary headers may end with -bin [info] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:445) [info] at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:972) [info] at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:966) [info] at io.grpc.Metadata$Key.of(Metadata.java:708) [info] at io.grpc.Metadata$Key.of(Metadata.java:704) [info] at org.apache.spark.sql.connect.client.SparkConnectClient$MetadataHeaderClientInterceptor$$anon$2.$anonfun$start$1(SparkConnectClient.scala:1112) [info] at org.apache.spark.sql.connect.client.SparkConnectClient$MetadataHeaderClientInterceptor$$anon$2.$anonfun$start$1$adapted(SparkConnectClient.scala:1106) [info] at scala.collection.immutable.Map$Map1.foreach(Map.scala:278) [info] at org.apache.spark.sql.connect.client.SparkConnectClient$MetadataHeaderClientInterceptor$$anon$2.start(SparkConnectClient.scala:1106) [info] at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:435) ``` ### Does this PR introduce _any_ user-facing change? Current behaviour: Fails for all header key-value pairs if the key has the `-bin` suffix with an `IllegalArgumentException`. New behaviour: Adds a `Metadata.BINARY_BYTE_MARSHALLER` header if the key has a `-bin` suffix and the value string is base64-encoded. ### How was this patch tested? Added a test to `SparkConnectClientSuite`. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
