wgtmac commented on code in PR #50926:
URL: https://github.com/apache/arrow/pull/50926#discussion_r3851607138
##########
cpp/src/arrow/csv/converter.cc:
##########
@@ -534,8 +534,9 @@ class NullConverter : public ConcreteConverter {
int32_t col_index) override {
NullBuilder builder(pool_);
- auto visit = [&](const uint8_t* data, uint32_t size, bool quoted) ->
Status {
- if (ARROW_PREDICT_TRUE(decoder_.IsNull(data, size, quoted))) {
+ auto visit = [&](const uint8_t* data, uint32_t size, bool quoted,
+ bool missing) -> Status {
+ if (missing || ARROW_PREDICT_TRUE(decoder_.IsNull(data, size, quoted))) {
Review Comment:
This can freeze an all-missing first block as a null-typed column in
StreamingReader. If a later block contains a value, the frozen NullConverter
rejects it. I'm not sure if this is a blocker but it seems to be better than
before?
##########
cpp/src/arrow/csv/options.h:
##########
@@ -61,6 +61,8 @@ struct ARROW_EXPORT ParseOptions {
bool ignore_empty_lines = true;
/// A handler function for rows which do not have the correct number of
columns
InvalidRowHandler invalid_row_handler;
+ /// Whether rows with fewer columns than expected are padded with nulls.
+ bool pad_short_rows = false;
Review Comment:
PyArrow's ParseOptions does not expose this new option
(constructor/property, pickle/equality, and Cython declaration). If PyArrow
should support it, please add the binding and tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
