AngersZhuuuu commented on a change in pull request #30973:
URL: https://github.com/apache/spark/pull/30973#discussion_r550953847
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala
##########
@@ -55,8 +58,11 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
override def outputPartitioning: Partitioning = child.outputPartitioning
override def doExecute(): RDD[InternalRow] = {
+ val hadoopConf = sqlContext.sessionState.newHadoopConf()
+ hadoopConf.set(SQLConf.SCRIPT_TRANSFORMATION_COMMAND_WRAPPER.key,
Review comment:
> Yes, it dose. It passes the configurations via local properties to
executor sides. should be able to access it via `SQLConf.get`.
This piece of code is not particularly familiar, I'm going to check this way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
