This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7568-f17eef1a0e03b08a35d5796d3ca41d9acea9d374 in repository https://gitbox.apache.org/repos/asf/texera.git
commit e9c6abac75f65f49021ab39eb5c9330bbf0b78c9 Author: Kary Zheng <[email protected]> AuthorDate: Fri Aug 14 08:49:19 2026 +0000 feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568) ### What changes were proposed in this PR? Schema inference and execution disagreed about what a blank cell is, and the schema side was the one that lost information. Inference set `nullValue("")` on its parser, so a blank read as an empty string: `tryParseDouble("")` fails, `tryParseBoolean("")` fails, and `inferField` lands on `tryParseString()`. One empty cell was enough to type a whole numeric column as STRING. Execution builds its parser without `nullValue`, so the same blank read as null there, which is what `AttributeTypeUtils.parseField` is written to pass through. The effect reaches well past the scan. Every downstream operator that does arithmetic on such a column then receives strings and fails on rows whose values are perfectly good numbers, not on the blank one. Hugging Face Iris Logistic Regression on a three-row file fails at the first row, where numpy is handed `array([['2.6', '0.75']], dtype='<U32')`. Dropping the setting leaves both sides reading a blank as null, and `tryParseDouble(null)` already answers DOUBLE, so the column keeps the type its values give it. One corner changes with it: a column that is blank in every sampled row now infers as INTEGER rather than STRING. Its values are null either way, so this renames the empty rather than reinterpreting anything. ### Any related issues, documentation, discussions? Closes #7550 ### How was this PR tested? `CSVScanSourceOpDescSpec` gains a case that writes a two-column file whose numeric column is blank on one row and asserts the inferred type is DOUBLE. It fails on the previous behavior, 16 passed / 1 failed before the change and 17 / 0 after. Since this touches inference every scan goes through, the whole module was run as well: 2187 passed, 0 failed. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Xinyuan Lin <[email protected]> --- .../source/scan/csv/CSVScanSourceOpDesc.scala | 5 ++++- .../source/scan/csv/CSVScanSourceOpDescSpec.scala | 24 ++++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala index 1a784a44ee..69088abb6c 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala @@ -107,7 +107,10 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc { csvSetting.setMaxColumns(maxColumns) csvSetting.setFormat(csvFormat) csvSetting.setHeaderExtractionEnabled(hasHeader) - csvSetting.setNullValue("") + // No setNullValue here, so a blank cell reads as null exactly as it does at + // execution time (CSVScanSourceOpExec builds its parser without one). Reading it + // as "" instead made inferField fall through to STRING, which typed a numeric + // column by its one empty cell rather than by its values. val parser = new CsvParser(csvSetting) parser.beginParsing(inputReader) diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala index 3049d7f02a..fe426552a9 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala @@ -97,6 +97,30 @@ class CSVScanSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter { opDesc.sourceSchema().getAttributes.map(_.getName).toList } + // Writes a numeric column with one blank cell and returns the absolute path. + private def writeCsvWithBlankNumericCell(): String = { + val tmpFile = Files.createTempFile("blank-cell-", ".csv") + tmpFile.toFile.deleteOnExit() + Files.write( + tmpFile, + "id,measure\n1,2.5\n2,\n3,4.5\n".getBytes(StandardCharsets.UTF_8) + ) + tmpFile.toString + } + + it should "infer a numeric column as DOUBLE even when one of its cells is blank" in { + val path = writeCsvWithBlankNumericCell() + csvScanSourceOpDesc.fileName = Some(path) + csvScanSourceOpDesc.setResolvedFileName(FileResolver.resolve(path)) + + // A blank used to read as "" while inferring and as null while executing. The "" + // fell through inferField to STRING, so one empty cell retyped the whole column + // and every downstream numeric operator then received strings. + assert( + csvScanSourceOpDesc.sourceSchema().getAttribute("measure").getType == AttributeType.DOUBLE + ) + } + it should "infer schema from single-line-data csv" in { parallelCsvScanSourceOpDesc.fileName = Some(TestOperators.CountrySalesSmallCsvPath)
