This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new 89d22b4b5c Recursive CTEs: Stage 1 - add config flag (#8828)
89d22b4b5c is described below
commit 89d22b4b5c62f26e6cc0fc86dfa631ada1f44567
Author: Matt Gapp <[email protected]>
AuthorDate: Wed Jan 17 13:24:20 2024 -0800
Recursive CTEs: Stage 1 - add config flag (#8828)
* add config flag for recursive ctes
update docs from script
update slt test for doc change
* restore testing pin
---
datafusion/common/src/config.rs | 5 +++++
datafusion/sql/src/query.rs | 13 ++++++++++++-
datafusion/sqllogictest/test_files/information_schema.slt | 2 ++
docs/source/user-guide/configs.md | 1 +
4 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 996a505dea..e00c179308 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -290,6 +290,11 @@ config_namespace! {
/// Hive. Note that this setting does not affect reading partitioned
/// tables (e.g. `/table/year=2021/month=01/data.parquet`).
pub listing_table_ignore_subdirectory: bool, default = true
+
+ /// Should DataFusion support recursive CTEs
+ /// Defaults to false since this feature is a work in progress and may
not
+ /// behave as expected
+ pub enable_recursive_ctes: bool, default = false
}
}
diff --git a/datafusion/sql/src/query.rs b/datafusion/sql/src/query.rs
index dd4cab1262..388377e3ee 100644
--- a/datafusion/sql/src/query.rs
+++ b/datafusion/sql/src/query.rs
@@ -54,7 +54,18 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
// Process CTEs from top to bottom
// do not allow self-references
if with.recursive {
- return not_impl_err!("Recursive CTEs are not supported");
+ if self
+ .context_provider
+ .options()
+ .execution
+ .enable_recursive_ctes
+ {
+ return plan_err!(
+ "Recursive CTEs are enabled but are not yet supported"
+ );
+ } else {
+ return not_impl_err!("Recursive CTEs are not supported");
+ }
}
for cte in with.cte_tables {
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt
b/datafusion/sqllogictest/test_files/information_schema.slt
index 44daa51416..b37b78ab6d 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -150,6 +150,7 @@ datafusion.execution.aggregate.scalar_update_factor 10
datafusion.execution.batch_size 8192
datafusion.execution.coalesce_batches true
datafusion.execution.collect_statistics false
+datafusion.execution.enable_recursive_ctes false
datafusion.execution.listing_table_ignore_subdirectory true
datafusion.execution.max_buffered_batches_per_output_file 2
datafusion.execution.meta_fetch_concurrency 32
@@ -225,6 +226,7 @@ datafusion.execution.aggregate.scalar_update_factor 10
Specifies the threshold f
datafusion.execution.batch_size 8192 Default batch size while creating new
batches, it's especially useful for buffer-in-memory batches since creating
tiny batches would result in too much metadata memory consumption
datafusion.execution.coalesce_batches true When set to true, record batches
will be examined between each operator and small batches will be coalesced into
larger batches. This is helpful when there are highly selective filters or
joins that could produce tiny output batches. The target batch size is
determined by the configuration setting
datafusion.execution.collect_statistics false Should DataFusion collect
statistics after listing files
+datafusion.execution.enable_recursive_ctes false Should DataFusion support
recursive CTEs Defaults to false since this feature is a work in progress and
may not behave as expected
datafusion.execution.listing_table_ignore_subdirectory true Should sub
directories be ignored when scanning directories for data files. Defaults to
true (ignores subdirectories), consistent with Hive. Note that this setting
does not affect reading partitioned tables (e.g.
`/table/year=2021/month=01/data.parquet`).
datafusion.execution.max_buffered_batches_per_output_file 2 This is the
maximum number of RecordBatches buffered for each output file being worked.
Higher values can potentially give faster write performance at the cost of
higher peak memory consumption
datafusion.execution.meta_fetch_concurrency 32 Number of files to read in
parallel when inferring schema and statistics
diff --git a/docs/source/user-guide/configs.md
b/docs/source/user-guide/configs.md
index 5e26e2b205..a812b74284 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -83,6 +83,7 @@ Environment variables are read during `SessionConfig`
initialisation so they mus
| datafusion.execution.soft_max_rows_per_output_file |
50000000 | Target number of rows in output files when writing
multiple. This is a soft max, so it can be exceeded slightly. There also will
be one file smaller than the limit if the total number of rows written is not
roughly divisible by the soft max
[...]
| datafusion.execution.max_buffered_batches_per_output_file | 2
| This is the maximum number of RecordBatches buffered
for each output file being worked. Higher values can potentially give faster
write performance at the cost of higher peak memory consumption
[...]
| datafusion.execution.listing_table_ignore_subdirectory |
true | Should sub directories be ignored when scanning
directories for data files. Defaults to true (ignores subdirectories),
consistent with Hive. Note that this setting does not affect reading
partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`).
[...]
+| datafusion.execution.enable_recursive_ctes |
false | Should DataFusion support recursive CTEs Defaults
to false since this feature is a work in progress and may not behave as
expected
[...]
| datafusion.optimizer.enable_distinct_aggregation_soft_limit |
true | When set to true, the optimizer will push a limit
operation into grouped aggregations which have no aggregate expressions, as a
soft limit, emitting groups once the limit is reached, before all rows in the
group are read.
[...]
| datafusion.optimizer.enable_round_robin_repartition |
true | When set to true, the physical plan optimizer will
try to add round robin repartitioning to increase parallelism to leverage more
CPU cores
[...]
| datafusion.optimizer.enable_topk_aggregation |
true | When set to true, the optimizer will attempt to
perform limit operations during aggregations, if possible
[...]
