roczei commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1339222364
Thanks @ocworld for the uploaded unit test! It works perfectly, it can
identify the issue.
Good:
```
- SPARK-35084: includes jars passed in through --packages in k8s client
driver mode
```
Bad:
```
- SPARK-35084: includes jars passed in through --packages in k8s client
driver mode *** FAILED ***
false was not equal to true (SparkSubmitSuite.scala:513)
```
Just for documentation. Here is my test case what I have validated in my
environment manually. I am using a spark-shell in a k8s Docker container.
It fails with the following error without your fix:
/tmp/spark.properties:
```
spark.kubernetes.submitInDriver=true
spark.kubernetes.authenticate.driver.serviceAccountName=spark
spark.kubernetes.namespace=default
spark.driver.blockManager.port=7079
spark.driver.port=7078
spark.blockManager.port=7079
spark.kubernetes.executor.label.name=executor
spark.kubernetes.driver.label.name=driver
spark.locality.wait=0
spark.executor.instances=1
spark.kubernetes.container.image=spark:spark-35084-upstream-no-fix
spark.master=k8s\://https\://kubernetes.default.svc.cluster.local\:443
spark.jars.packages=com.github.music-of-the-ainur\:almaren-framework_2.12\:0.9.4-3.2,com.github.music-of-the-ainur\:http-almaren_2.12\:1.2.4-3.2
spark.driver.host=172.17.0.4
spark.kubernetes.driver.pod.name=spark-submitter-spark-35084-upstream-no-fix-rxpjs
```
```
spark-shell --properties-file /tmp/spark.properties
...
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 17.0.5)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
<console>:22: error: object music is not a member of package com.github
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
^
scala> import com.github.music.of.the.ainur.almaren.Almaren
<console>:22: error: object music is not a member of package com.github
import com.github.music.of.the.ainur.almaren.Almaren
^
scala> import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.DataFrame
scala> val almaren = Almaren("App Name")
<console>:26: error: not found: value Almaren
val almaren = Almaren("App Name")
^
scala> $intp.isettings.maxPrintString = 0
$intp.isettings.maxPrintString: Int = 0
scala> spark.conf.get("spark.jars")
res0: String = ""
```
and this is the good one which includes your fix:
/tmp/spark.properties
```
spark.kubernetes.submitInDriver=true
spark.kubernetes.authenticate.driver.serviceAccountName=spark
spark.kubernetes.namespace=default
spark.driver.blockManager.port=7079
spark.driver.port=7078
spark.blockManager.port=7079
spark.kubernetes.executor.label.name=executor
spark.kubernetes.driver.label.name=driver
spark.locality.wait=0
spark.executor.instances=1
spark.kubernetes.container.image=spark:spark-35084-upstream-with-fix
spark.master=k8s\://https\://kubernetes.default.svc.cluster.local\:443
spark.jars.packages=com.github.music-of-the-ainur\:almaren-framework_2.12\:0.9.4-3.2,com.github.music-of-the-ainur\:http-almaren_2.12\:1.2.4-3.2
spark.driver.host=172.17.0.3
spark.kubernetes.driver.pod.name=spark-submitter-spark-35084-upstream-with-fix-whxzl
```
```
spark-shell --properties-file /tmp/spark.properties
...
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 17.0.5)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
scala> import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.Almaren
scala> import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.DataFrame
scala> val almaren = Almaren("App Name")
almaren: com.github.music.of.the.ainur.almaren.Almaren.type =
com.github.music.of.the.ainur.almaren.Almaren$@4c2f971
scala> $intp.isettings.maxPrintString = 0
$intp.isettings.maxPrintString: Int = 0
scala> spark.conf.get("spark.jars")
res0: String =
file:///home/sparkuser/.ivy2/jars/com.github.music-of-the-ainur_almaren-framework_2.12-0.9.4-3.2.jar,file:///home/sparkuser/.ivy2/jars/com.github.music-of-the-ainur_http-almaren_2.12-1.2.4-3.2.jar,file:///home/sparkuser/.ivy2/jars/com.typesafe.scala-logging_scala-logging_2.12-3.9.4.jar,file:///home/sparkuser/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.2.1.jar,file:///home/sparkuser/.ivy2/jars/com.databricks_spark-xml_2.12-0.14.0.jar,file:///home/sparkuser/.ivy2/jars/com.github.music-of-the-ainur_quenya-dsl_2.12-1.2.0-3.2.jar,file:///home/sparkuser/.ivy2/jars/org.scala-lang_scala-reflect-2.12.13.jar,file:///home/sparkuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.30.jar,file:///home/sparkuser/.ivy2/jars/org.tukaani_xz-1.8.jar,file:///home/sparkuser/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar,file:///home/sparkuser/.ivy2/jars/commons-io_commons-io-2.8.0.jar,file:///home/sparkuser/.ivy2/jars/org.glassfish.jaxb_txw2-2.3.4.jar,file:///home/sparkuser/.ivy2/jars/org.apa
che.ws.xmlschema_xmlschema-core-2.2.5.jar,file:///home/sparkuser/.ivy2/jars/com.lihaoyi_requests_2.12-0.7.0.jar,file:///home/sparkuser/.ivy2/jars/com.lihaoyi_geny_2.12-0.6.10.jar
scala>
```
@dongjoon-hyun, @HyukjinKwon, @holdenk Could you please review / approve
this pull request? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
