Copilot commented on code in PR #50483:
URL: https://github.com/apache/arrow/pull/50483#discussion_r3566512059
##########
cpp/src/arrow/csv/parser_test.cc:
##########
@@ -936,5 +936,45 @@ TEST(BlockParser, RowNumberAppendedToError) {
}
}
+TEST(BlockParser, EmbeddedNulInQuotedFieldAfterBulkFilterActivates) {
+ // Regression test for GH-50481. Once the running average value length
+ // crosses the bulk filter's activation threshold, a NUL byte embedded in a
+ // quoted field could hide the real closing quote if both landed in the
+ // same 8-byte SIMD word, causing the parser to keep consuming subsequent
+ // bytes as if still inside the quoted field.
+ constexpr int32_t num_cols = 64;
+ constexpr int32_t num_filler_rows = 512; // = kTargetChunkSize / num_cols
+
+ std::string csv;
+ for (int32_t r = 0; r < num_filler_rows; ++r) {
+ for (int32_t c = 0; c < num_cols; ++c) {
+ if (c) csv += ',';
+ // 12 bytes/value, well above the bulk filter's 10-bytes/value
+ // activation threshold.
+ csv += "xxxxxxxxxxxx";
+ }
+ csv += '\n';
+ }
+ // This row's first field carries a real embedded NUL byte immediately
+ // before its closing quote - the byte pattern a misaligned
+ // implicit-length SIMD scan can hide.
+ csv += "\"abc";
+ csv += '\0';
+ csv += "def\"";
+ for (int32_t c = 1; c < num_cols; ++c) {
+ csv += ",xxxxxxxxxxxx";
+ }
+ csv += '\n';
+
+ BlockParser parser(ParseOptions::Defaults());
Review Comment:
This test currently constructs `BlockParser` with `num_cols=-1`, which
triggers the 1-row "infer number of columns" parse path. With 64 columns ×
12-byte values, that first-row inference is already enough to enable
`use_bulk_filter_` before the 512 filler rows are processed, so the test
doesn't actually exercise the intended "bulk filter activates after enough
rows" transition described in the comment/name. Constructing the parser with a
known column count makes the bulk-filter activation depend on the filler
rows/chunking as intended and keeps the regression aligned with GH-50481's
trigger condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
