turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path:
${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
Review Comment:
Tested with yarn-cluster mode:
```
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's
query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE,
statement:
import org.apache.iceberg._
24/04/01 20:12:36 INFO OperationAuditLogger:
operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala
state=RUNNING user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala:
Spark application name:
kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
application ID: application_1710906451321_1347246
application web UI:
https://apollo-rno-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1710906451321_1347246,https://apollo-rno-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1710906451321_1347246
master: yarn
deploy mode: cluster
version: 3.1.1.1.3.0
Start time: 2024-04-01T20:11:29.244
User: b_stf
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's
query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE,
time taken: 0.154 seconds
24/04/01 20:12:36 INFO OperationAuditLogger:
operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala
state=FINISHED user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala:
statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms,
operationCpuTime=0 ms
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement:
Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE
-> FINISHED_STATE, time taken: 0.197 seconds
+------------------------------+
| output |
+------------------------------+
| import org.apache.iceberg._ |
+------------------------------+
1 row selected (6.231 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
