alamb commented on code in PR #13981:
URL: https://github.com/apache/datafusion/pull/13981#discussion_r1900827004
##########
datafusion-examples/README.md:
##########
@@ -54,6 +54,7 @@ cargo run --example dataframe
- [`catalog.rs`](examples/catalog.rs): Register the table into a custom catalog
- [`composed_extension_codec`](examples/composed_extension_codec.rs): Example
of using multiple extension codecs for serialization / deserialization
- [`csv_sql_streaming.rs`](examples/csv_sql_streaming.rs): Build and run a
streaming query plan from a SQL statement against a local CSV file
+- [`csv_json_opener.rs`](examples/csv_json_opener.rs): Demonstrate a scanning
against an Arrow data source (CSV/JSON) and fetching results
Review Comment:
```suggestion
- [`csv_json_opener.rs`](examples/csv_json_opener.rs): Use low level
`FileOpener` APIs to read CSV/JSON into Arrow `RecordBatch`es
```
##########
datafusion-examples/examples/csv_json_opener.rs:
##########
@@ -15,28 +15,34 @@
// specific language governing permissions and limitations
// under the License.
-use std::{sync::Arc, vec};
+use std::sync::Arc;
+use arrow_schema::{DataType, Field, Schema};
use datafusion::{
assert_batches_eq,
datasource::{
file_format::file_compression_type::FileCompressionType,
listing::PartitionedFile,
object_store::ObjectStoreUrl,
- physical_plan::{CsvConfig, CsvOpener, FileScanConfig, FileStream},
+ physical_plan::{CsvConfig, CsvOpener, FileScanConfig, FileStream,
JsonOpener},
},
error::Result,
physical_plan::metrics::ExecutionPlanMetricsSet,
test_util::aggr_test_schema,
};
-
use futures::StreamExt;
-use object_store::local::LocalFileSystem;
+use object_store::{local::LocalFileSystem, memory::InMemory, ObjectStore};
-/// This example demonstrates a scanning against an Arrow data source (CSV) and
+/// This example demonstrates a scanning against an Arrow data source
(CSV/JSON) and
Review Comment:
```suggestion
/// This example demonstrates using the low level [`FileStream`] /
[`FileOpener`] APIs to directly
/// read data from (CSV/JSON) into Arrow RecordBatches.
///
/// If you want to query data in CSV or JSON files, see the [`dataframe.rs`]
and [`sql_query.rs`] examples
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
