comphead commented on code in PR #6112:
URL: https://github.com/apache/datafusion-comet/pull/6112#discussion_r4417473501
##########
.github/workflows/pr_build_linux.yml:
##########
@@ -761,26 +763,37 @@ jobs:
restore-keys: |
${{ runner.os }}-java-maven-
- - name: Cache TPC-H data
+ - name: Restore TPC-H data
id: cache-tpch
- uses: actions/cache@v6
+ uses: actions/cache/restore@v6
with:
path: ./tpch
- key: tpch-${{ hashFiles('.github/workflows/pr_build_linux.yml') }}
+ key: tpch-sf${{ env.TPC_SCALE_FACTOR }}-p${{ env.TPC_NUM_PARTITIONS
}}-${{ hashFiles('spark/src/test/scala/org/apache/spark/sql/GenTPCHData.scala')
}}
Review Comment:
The key hashes only `GenTPCHData.scala`, but the generated data also depends
on `TPCH.scala`, which holds `Dbgen` (it builds the `./dbgen -q ... -T <table>
-s <sf>` command line) and `TPCHTables` (the schemas). A change there would
still hit the cache and run the queries on the old dataset, which is the
failure this PR removes. Could the key hash `TPCH.scala` too, here and in the
save key at line 796? For TPC-DS, `GenTPCDSData` and `TPCDSTables` come from
Spark's `spark-sql` tests jar at the default profile's `spark.version`, so a
Spark bump does not rotate that key either. Is that intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
