advancedxy commented on a change in pull request #25616: [SPARK-28907][CORE]
Review invalid usage of new Configuration()
URL: https://github.com/apache/spark/pull/25616#discussion_r320011857
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFileWholeTextReader.scala
##########
@@ -45,6 +45,7 @@ class HadoopFileWholeTextReader(file: PartitionedFile, conf:
Configuration)
val attemptId = new TaskAttemptID(new TaskID(new JobID(), TaskType.MAP,
0), 0)
val hadoopAttemptContext = new TaskAttemptContextImpl(conf, attemptId)
val reader = new WholeTextFileRecordReader(fileSplit,
hadoopAttemptContext, 0)
+ reader.setConf(hadoopAttemptContext.getConfiguration)
Review comment:
Some tests in `WholeTextFileSuite` and `SaveLoadSuite` are failing without
this change.
However, the failure is introduced by my change to
`WholeTextFileRecordReader`
https://github.com/apache/spark/blob/149de72c220cbc094f0b8756c535cf1bd796a48e/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala#L70-L73
We use `getConf` instead of `new Configuration`, then should call `setConf`
first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
