westonpace commented on code in PR #13782:
URL: https://github.com/apache/arrow/pull/13782#discussion_r975585346
##########
cpp/src/arrow/dataset/dataset.h:
##########
@@ -37,17 +38,91 @@ namespace dataset {
using RecordBatchGenerator =
std::function<Future<std::shared_ptr<RecordBatch>>()>;
+/// \brief Description of a column to scan
+struct FragmentSelectionColumn {
+ /// \brief The path to the column to load
+ FieldPath path;
+ /// \brief The type of the column in the dataset schema
+ ///
+ /// A format may choose to ignore this field completely. For example, when
+ /// reading from IPC the reader can just return the column in the data type
+ /// that is stored on disk. There is no point in doing anything special.
+ ///
+ /// However, some formats may be capable of casting on the fly. For example,
+ /// when reading from CSV, if we know the target type of the column, we can
+ /// convert from string to the target type as we read.
+ DataType* requested_type;
+ /// \brief The index in the output selection of this column
+ int selection_index;
+};
+/// \brief Instructions for scanning a particular fragment
+///
+/// The fragment scan request is dervied from ScanV2Options. The main
+/// difference is that the scan options are based on the dataset schema
+/// while the fragment request is based on the fragment schema.
+struct FragmentScanRequest {
+ /// \brief A row filter
+ ///
+ /// The filter expression should be written against the fragment schema.
+ ///
+ /// \see ScanV2Options for details on how this filter should be applied
+ compute::Expression filter = compute::literal(true);
+
+ /// \brief The columns to scan
+ ///
+ /// These indices refer to the fragment schema
+ ///
+ /// Note: This is NOT a simple list of top-level column indices.
+ /// For more details \see ScanV2Options
+ ///
+ /// If possible a fragment should only read from disk the data needed
+ /// to satisfy these columns. If a format cannot partially read a nested
+ /// column (e.g. JSON) then it must apply the column selection (in memory)
+ /// before returning the scanned batch.
+ std::vector<FragmentSelectionColumn> columns;
+ /// \brief Options specific to the format being scanned
+ FragmentScanOptions* format_scan_options;
+};
+
+class FragmentScanner {
+ public:
+ /// This instance will only be destroyed after all ongoing scan futures
+ /// have been completed.
+ ///
+ /// This means any callbacks created as part of the scan can safely
+ /// capture `this`
+ virtual ~FragmentScanner() = default;
+ /// \brief Scan a batch of data from the file
+ /// \param batch_number The index of the batch to read
+ virtual Future<std::shared_ptr<RecordBatch>> ScanBatch(int batch_number) = 0;
+ /// \brief Calculate an estimate of how many data bytes the given batch will
represent
+ ///
+ /// "Data bytes" should be the total size of all the buffers once the data
has been
+ /// decoded into the Arrow format.
+ virtual int64_t EstimatedDataBytes(int batch_number) = 0;
+ /// \brief The number of batches in the fragment to scan
+ virtual int NumBatches() = 0;
+};
+
+struct InspectedFragment {
+ explicit InspectedFragment(std::vector<std::string> column_names)
+ : column_names(std::move(column_names)) {}
+ std::vector<std::string> column_names;
Review Comment:
My original thinking was that, with CSV, we can't know the types at this
point. Plus, for the basic evolution strategy, we don't need to know the
types. With that strategy all we need to know is the column names.
However, looking at this with fresh eyes, I think we do know the types with
CSV. We don't re-infer with each new fragment and we assert that the schema
matches the dataset schema (you cannot do a type-widening evolution with CSV
for example).
In that case maybe I can get rid of `InspectedFragment` entirely.
The other reason for `InspectedFragment` was for the case where we know more
than types+names. For example, we might know the parquet column ids for each
column. However, we should be able to encode that information in a schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
