appletreeisyellow commented on code in PR #7005:
URL: https://github.com/apache/arrow-datafusion/pull/7005#discussion_r1269750623
##########
benchmarks/bench.sh:
##########
@@ -318,6 +338,62 @@ run_sort() {
$CARGO_COMMAND --bin parquet -- sort --path "${DATA_DIR}" --scale-factor
1.0 --iterations 10 -o ${RESULTS_FILE}
}
+
+# Downloads the single file hits.parquet ClickBench datasets from
+# https://github.com/ClickHouse/ClickBench/tree/main#data-loading
+#
+# Creates data in $DATA_DIR/hits.parquet
+data_clickbench_1() {
+ pushd "${DATA_DIR}" > /dev/null
+
+ # Avoid downloading if it already exists and is the right size
+ OUTPUT_SIZE=`wc -c hits.parquet 2>/dev/null | awk '{print $1}' || true`
+ echo -n "Checking hits.parquet..."
+ if test "${OUTPUT_SIZE}" = "14779976446"; then
+ echo -n "... found ${OUTPUT_SIZE} bytes ..."
+ else
+ URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet";
+ echo -n "... downloading ${URL} (14GB) ... "
Review Comment:
1. Do you mean the other way? Since `OUTPUT_SIZE` is constant in the if
statement and dynamic in the else statement
```suggestion
if test "${OUTPUT_SIZE}" = "14779976446"; then
echo -n "... downloading ${URL} (14GB) ... "
else
URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet";
echo -n "... found ${OUTPUT_SIZE} bytes ..."
```
2. Just curious, why the unit is `byte` in line 353 and `KB` in line 376?
Should they be the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
