alamb commented on code in PR #23315:
URL: https://github.com/apache/datafusion/pull/23315#discussion_r3522940778
##########
benchmarks/README.md:
##########
@@ -496,6 +496,52 @@ The ClickBench[1] benchmarks are widely cited in the
industry and
focus on grouping / aggregation / filtering. This runner uses the
scripts and queries from [2].
+The runner applies two ClickBench-specific setup steps automatically:
+
+- ClickBench stores `EventDate` as `UInt16` days since `1970-01-01`.
+ The runner registers the parquet data as `hits_raw`, then creates a
+ `hits` view that casts `EventDate` through `INTEGER` to `DATE` for the
+ benchmark queries.
+- The full and partitioned ClickBench datasets may store string columns
Review Comment:
This only applies to the partitioned ClickBench dataset -- what happens is
that the string columns don't have a "string" logical type annotation in the
parquet files
Maybe a better description is:
The source partitioned ClickBench datasets has string columns without
the "string" Parquet logical type annotation. These must be treated as
strings to correctly run the query, so the runner enables the parquet
`binary_as_string`
option.
##########
benchmarks/README.md:
##########
@@ -496,6 +496,52 @@ The ClickBench[1] benchmarks are widely cited in the
industry and
focus on grouping / aggregation / filtering. This runner uses the
scripts and queries from [2].
+The runner applies two ClickBench-specific setup steps automatically:
+
+- ClickBench stores `EventDate` as `UInt16` days since `1970-01-01`.
+ The runner registers the parquet data as `hits_raw`, then creates a
+ `hits` view that casts `EventDate` through `INTEGER` to `DATE` for the
+ benchmark queries.
+- The full and partitioned ClickBench datasets may store string columns
+ as binary values. The runner enables the parquet `binary_as_string`
+ option so those columns are read as strings. Small test subsets may
+ already have string columns and not need this option.
+
+If you set up ClickBench manually through SQL, use the same pattern:
+
+```sql
+CREATE EXTERNAL TABLE hits_raw
+STORED AS PARQUET
+LOCATION 'benchmarks/data/hits.parquet'
Review Comment:
I m pretty sure this is only necessary for hits_partititoned
##########
benchmarks/README.md:
##########
@@ -496,6 +496,52 @@ The ClickBench[1] benchmarks are widely cited in the
industry and
focus on grouping / aggregation / filtering. This runner uses the
scripts and queries from [2].
+The runner applies two ClickBench-specific setup steps automatically:
+
+- ClickBench stores `EventDate` as `UInt16` days since `1970-01-01`.
+ The runner registers the parquet data as `hits_raw`, then creates a
+ `hits` view that casts `EventDate` through `INTEGER` to `DATE` for the
+ benchmark queries.
+- The full and partitioned ClickBench datasets may store string columns
+ as binary values. The runner enables the parquet `binary_as_string`
+ option so those columns are read as strings. Small test subsets may
+ already have string columns and not need this option.
+
+If you set up ClickBench manually through SQL, use the same pattern:
+
+```sql
+CREATE EXTERNAL TABLE hits_raw
+STORED AS PARQUET
+LOCATION 'benchmarks/data/hits.parquet'
+OPTIONS ('binary_as_string' 'true');
+
+CREATE VIEW hits AS
+SELECT * EXCEPT ("EventDate"),
+ CAST(CAST("EventDate" AS INTEGER) AS DATE) AS "EventDate"
+FROM hits_raw;
+```
+
+From the repository root, download data and run the default ClickBench
+queries against the single parquet file:
+
+```shell
+./benchmarks/bench.sh data clickbench_1
+cargo run --release --bin dfbench -- clickbench \
Review Comment:
I don't think anyone would run a command like that -- instead if they want
to run all the queries they would use `bench.sh run`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
