westonpace commented on issue #34346:
URL: https://github.com/apache/arrow/issues/34346#issuecomment-1478612255
@dinimar Sorry, the field I was thinking of was
`arrow::csv::ConvertOptions::include_columns`. It will still need to be
possible to ask for all columns (and this should still be the default).
One way to do this would be to change `include_columns` from
`std::vector<std::string>` to `std::optional<std::vector<std::string>>` with
the default being `std::nullopt` which should be interpreted as "read all
columns". However, this would be a backwards incompatible change.
Another way to do this would be to introduce a new field `bool
allow_no_columns` which defaults to `false` with the semantics:
* If `include_columns` is empty and `allow_no_columns` is false then read
all columns
* If `include_columns` is empty and `allow_no_columns` is true then don't
read any columns
* If `include_columns` is not empty then ignore `allow_no_columns`.
This is backwards compatible but more confusing and redundant.
A final option, for best performance, is to create an entirely different CSV
reader implementation (e.g. similar to `StreamingReader` and `TableReader`)
which simplifies parsing by only searching and counting newlines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
