kosiew commented on PR #15295: URL: https://github.com/apache/datafusion/pull/15295#issuecomment-2864869245
> there's 10 v1 files, 5 v2 files, 3 v3 files and 1 v4 files. Ideally ListingTableConfig could just derive the mapping from each. Is that possible with your abstraction? # Here's the summary of the diff ## v2 vs v1 The v1 schema had 9 fields. The v2 schema now has 10 fields, with the additional query_params field providing more structured metadata for capturing customer-specific identifiers within the URL query parameters. ## v3 vs v2 The v3 schema now has 11 fields, with the addition of the error field, which provides a more complete picture of API request outcomes, particularly for capturing error details. ## v4 vs v3 Field Count: Remains 11 overall, but the nested query_params struct grew from 1 to 4 fields. Can you try reading all the files with `ListingTableConfig` that uses table_schema, adapter_factory like this excerpt (from [example](https://github.com/apache/datafusion/pull/15295#issuecomment-2750776919))? ```rust let adapter_factory: Arc<dyn SchemaAdapterFactory> = Arc::new(NestedStructSchemaAdapterFactory {}); ..... ListingTableConfig .... .with_schema(table_schema.as_ref().clone().into()) .with_schema_adapter_factory(adapter_factory); ``` where table_schema = v4? With this ListingTableConfig, it will read all files (v1, v2, v3, v4) and adapt the fields to v4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
