[ https://issues.apache.org/jira/browse/SPARK-25134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-25134: --------------------------------- Affects Version/s: (was: 2.3.1) 2.4.0 > Csv column pruning with checking of headers throws incorrect error > ------------------------------------------------------------------ > > Key: SPARK-25134 > URL: https://issues.apache.org/jira/browse/SPARK-25134 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.0 > Environment: spark master branch at > a791c29bd824adadfb2d85594bc8dad4424df936 > Reporter: koert kuipers > Priority: Minor > > hello! > seems to me there is some interaction between csv column pruning and the > checking of csv headers that is causing issues. for example this fails: > {code:scala} > Seq(("a", "b")).toDF("columnA", "columnB").write > .format("csv") > .option("header", true) > .save(dir) > spark.read > .format("csv") > .option("header", true) > .option("enforceSchema", false) > .load(dir) > .select("columnA") > .show > {code} > the error is: > {code:bash} > 291.0 (TID 319, localhost, executor driver): > java.lang.IllegalArgumentException: Number of column in CSV header is not > equal to number of fields in the schema: > [info] Header length: 1, schema size: 2 > {code} > if i remove the project it works fine. if i disable column pruning it also > works fine. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org