Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15813#discussion_r87121887
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
---
@@ -210,14 +196,21 @@ class CSVFileFormat extends TextBasedFileFormat with
DataSourceRegister {
private def readText(
sparkSession: SparkSession,
options: CSVOptions,
- location: String): RDD[String] = {
+ inputPaths: Seq[String]): Dataset[String] = {
if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
- sparkSession.sparkContext.textFile(location)
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = inputPaths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
--- End diff --
Hi @JoshRosen, I just happened to look at this one and I am just curious.
IIUC, the schema from the `sparkSession.baseRelationToDataFrame` will always
has only `value` column not including partitioned columns (it is empty and also
`inputPaths` will be always left files).
So, my question is, is that `.select("value")` used just to doubly make
sure? Just curious.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
