Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/19224#discussion_r161742050
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala
---
@@ -178,21 +166,15 @@ object MultiLineJsonDataSource extends JsonDataSource
{
file: PartitionedFile,
parser: JacksonParser,
schema: StructType): Iterator[InternalRow] = {
- def partitionedFileString(ignored: Any): UTF8String = {
- Utils.tryWithResource {
- CodecStreams.createInputStreamWithCloseResource(conf,
file.filePath)
--- End diff --
I can see that at the safe parser construction the UTF8String.fromString()
will be always used but I am wondering whether
CodecStreams.createInputStreamWithCloseResource() functionality is got lost. As
earlier this was the default to be tried and UTF8String.fromBytes was the
fallback.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
