westhide commented on PR #15335: URL: https://github.com/apache/datafusion/pull/15335#issuecomment-2740991253
> Thank you @westhide > > Should we remove the `batch_size` from JSON source too? > > https://github.com/apache/datafusion/blob/dd9c3a815d7b4af2ef503ea557332ecc700af318/datafusion/datasource-json/src/source.rs#L254 Can we add `context: Arc<TaskContext>` to the `create_file_opener` args? The default `batch_size` is from `context.session_config().batch_size()`. If we remove `batch_size` property in `JsonSource`, we need to get the default value by `context` ```rust pub trait FileSource: Send + Sync { /// Creates a `dyn FileOpener` based on given parameters fn create_file_opener( &self, object_store: Arc<dyn ObjectStore>, base_config: &FileScanConfig, partition: usize, ) -> Arc<dyn FileOpener>; // ... } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
