Maxim Gekk created SPARK-26007: ---------------------------------- Summary: DataFrameReader.csv() should respect to spark.sql.columnNameOfCorruptRecord Key: SPARK-26007 URL: https://issues.apache.org/jira/browse/SPARK-26007 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.0 Reporter: Maxim Gekk
The csv() method of DataFrameReader doesn't take into account the SQL config spark.sql.columnNameOfCorruptRecord while creating an instance of CSVOptions: https://github.com/apache/spark/blob/2d085c13b7f715dbff23dd1f81af45ff903d1a79/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L491-L494 This should be fixed by passing sparkSession.sessionState.conf.columnNameOfCorruptRecord as a constructor parameter to CSVOptions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org