appletreeisyellow commented on code in PR #7005:
URL: https://github.com/apache/arrow-datafusion/pull/7005#discussion_r1269881472
##########
benchmarks/bench.sh:
##########
@@ -318,6 +338,62 @@ run_sort() {
$CARGO_COMMAND --bin parquet -- sort --path "${DATA_DIR}" --scale-factor
1.0 --iterations 10 -o ${RESULTS_FILE}
}
+
+# Downloads the single file hits.parquet ClickBench datasets from
+# https://github.com/ClickHouse/ClickBench/tree/main#data-loading
+#
+# Creates data in $DATA_DIR/hits.parquet
+data_clickbench_1() {
+ pushd "${DATA_DIR}" > /dev/null
+
+ # Avoid downloading if it already exists and is the right size
+ OUTPUT_SIZE=`wc -c hits.parquet 2>/dev/null | awk '{print $1}' || true`
+ echo -n "Checking hits.parquet..."
+ if test "${OUTPUT_SIZE}" = "14779976446"; then
+ echo -n "... found ${OUTPUT_SIZE} bytes ..."
+ else
+ URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet";
+ echo -n "... downloading ${URL} (14GB) ... "
Review Comment:
> This means that you can run `bench.sh download clickbench_1` and be sure
you have the most recent data but not have to download all 14GB if it already
exists
Ah, got it! Thank you for the explanation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
