This is an automated email from the ASF dual-hosted git repository.
comphead pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new c7dbfeb79a Minor: Improve documentation for `catalog.has_header`
config option (#10452)
c7dbfeb79a is described below
commit c7dbfeb79a0f41b6098184de33499546697ef631
Author: Andrew Lamb <[email protected]>
AuthorDate: Mon May 13 10:49:27 2024 -0400
Minor: Improve documentation for `catalog.has_header` config option (#10452)
* Minor: document catalog.has_header better
* update docs
* update test
---
datafusion/common/src/config.rs | 3 ++-
datafusion/sqllogictest/test_files/information_schema.slt | 2 +-
docs/source/user-guide/configs.md | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index c60f843393..0f1d9b8f02 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -181,7 +181,8 @@ config_namespace! {
/// Type of `TableProvider` to use when loading `default` schema
pub format: Option<String>, default = None
- /// If the file has a header
+ /// Default value for `format.has_header` for `CREATE EXTERNAL TABLE`
+ /// if not specified explicitly in the statement.
pub has_header: bool, default = false
}
}
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt
b/datafusion/sqllogictest/test_files/information_schema.slt
index de00cf9d05..6f31973fdb 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -246,7 +246,7 @@ datafusion.catalog.create_default_catalog_and_schema true
Whether the default ca
datafusion.catalog.default_catalog datafusion The default catalog name - this
impacts what SQL queries use if not specified
datafusion.catalog.default_schema public The default schema name - this
impacts what SQL queries use if not specified
datafusion.catalog.format NULL Type of `TableProvider` to use when loading
`default` schema
-datafusion.catalog.has_header false If the file has a header
+datafusion.catalog.has_header false Default value for `format.has_header` for
`CREATE EXTERNAL TABLE` if not specified explicitly in the statement.
datafusion.catalog.information_schema true Should DataFusion provide access to
`information_schema` virtual tables for displaying schema information
datafusion.catalog.location NULL Location scanned to load tables for `default`
schema
datafusion.execution.aggregate.scalar_update_factor 10 Specifies the threshold
for using `ScalarValue`s to update accumulators during high-cardinality
aggregations for each input batch. The aggregation is considered
high-cardinality if the number of affected groups is greater than or equal to
`batch_size / scalar_update_factor`. In such cases, `ScalarValue`s are utilized
for updating accumulators, rather than the default batch-slice approach. This
can lead to performance improvements. By [...]
diff --git a/docs/source/user-guide/configs.md
b/docs/source/user-guide/configs.md
index ef2a2a4119..0cfd81eff7 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -43,7 +43,7 @@ Environment variables are read during `SessionConfig`
initialisation so they mus
| datafusion.catalog.information_schema |
false | Should DataFusion provide access to
`information_schema` virtual tables for displaying schema information
[...]
| datafusion.catalog.location |
NULL | Location scanned to load tables for `default`
schema
[...]
| datafusion.catalog.format |
NULL | Type of `TableProvider` to use when loading
`default` schema
[...]
-| datafusion.catalog.has_header |
false | If the file has a header
[...]
+| datafusion.catalog.has_header |
false | Default value for `format.has_header` for `CREATE
EXTERNAL TABLE` if not specified explicitly in the statement.
[...]
| datafusion.execution.batch_size |
8192 | Default batch size while creating new batches, it's
especially useful for buffer-in-memory batches since creating tiny batches
would result in too much metadata memory consumption
[...]
| datafusion.execution.coalesce_batches |
true | When set to true, record batches will be examined
between each operator and small batches will be coalesced into larger batches.
This is helpful when there are highly selective filters or joins that could
produce tiny output batches. The target batch size is determined by the
configuration setting
[...]
| datafusion.execution.collect_statistics |
false | Should DataFusion collect statistics after listing
files
[...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
