westonpace commented on code in PR #14663:
URL: https://github.com/apache/arrow/pull/14663#discussion_r1032886169
##########
cpp/src/arrow/dataset/file_base.cc:
##########
@@ -108,6 +131,18 @@ Future<std::optional<int64_t>> FileFormat::CountRows(
return Future<std::optional<int64_t>>::MakeFinished(std::nullopt);
}
+Future<std::shared_ptr<InspectedFragment>> FileFormat::InspectFragment(
+ const FileSource& source, const FragmentScanOptions* format_options,
+ compute::ExecContext* exec_context) const {
+ return Status::NotImplemented("The new scanner is not yet supported on this
format");
Review Comment:
I changed this around to read "scan2 node". Although I hope we don't end up
needing a scan2 node and this can just replace the scan node.
##########
cpp/src/arrow/dataset/file_csv.cc:
##########
@@ -52,9 +53,99 @@ using internal::SerialExecutor;
namespace dataset {
+struct CsvInspectedFragment : public InspectedFragment {
+ CsvInspectedFragment(std::vector<std::string> column_names,
+ std::shared_ptr<io::InputStream> input_stream, int64_t
num_bytes)
+ : InspectedFragment(std::move(column_names)),
+ input_stream(std::move(input_stream)),
+ num_bytes(num_bytes) {}
+ // We need to start reading the file in order to figure out the column names
and
+ // so we save off the input stream
+ std::shared_ptr<io::InputStream> input_stream;
+ int64_t num_bytes;
+};
+
+class CsvFileScanner : public FragmentScanner {
+ public:
+ CsvFileScanner(std::shared_ptr<csv::StreamingReader> reader, int num_batches,
+ int64_t best_guess_bytes_per_batch)
+ : reader_(std::move(reader)),
+ num_batches_(num_batches),
+ best_guess_bytes_per_batch_(best_guess_bytes_per_batch) {}
+
+ Future<std::shared_ptr<RecordBatch>> ScanBatch(int batch_number) override {
+ // We should be called in increasing order but let's verify that in case
it changes.
Review Comment:
Yes. Thanks. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
