kaxil commented on code in PR #73527:
URL: https://github.com/apache/airflow/pull/73527#discussion_r4070483843
##########
.github/actions/install-prek/action.yml:
##########
@@ -143,16 +149,28 @@ runs:
shell: bash
run: cat ~/.cache/prek/prek.log || true
if: always()
+ # A pull request that restored the cache has nothing new to save: the
stash it got (usually the
+ # one from the base branch) is what it would upload again, and
re-compressing and uploading
+ # ~7 GB costs about five minutes per run. Save on a miss, and on non-PR
runs (the canary on
+ # the default branch) so the base-branch stash stays fresh within its
retention window.
- name: "Prepare .tar file from prek cache"
run: |
- tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ if command -v pigz > /dev/null 2>&1; then
+ tar -C ~ -cf - .cache/prek | pigz > /tmp/cache-prek.tar.gz
+ else
+ tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ fi
shell: bash
- if: inputs.save-cache == 'true'
+ if: >
+ inputs.save-cache == 'true'
+ && (steps.restore-prek-tar.outputs.tar-restored != 'true' ||
github.event_name != 'pull_request')
- name: "Save prek cache"
uses:
apache/infrastructure-actions/stash/save@61dcea11f19e2bbe1263f14d72235e8da17d3ad0
# save/v1.0.0
with:
key: ${{ steps.cache-key.outputs.key }}
path: /tmp/cache-prek.tar.gz
if-no-files-found: 'error'
retention-days: '2'
Review Comment:
Applied in 57bbdb2, thanks. Confirmed the pinned stash action forwards it to
upload-artifact with a default of 6.
##########
.github/actions/install-prek/action.yml:
##########
@@ -143,16 +149,28 @@ runs:
shell: bash
run: cat ~/.cache/prek/prek.log || true
if: always()
+ # A pull request that restored the cache has nothing new to save: the
stash it got (usually the
+ # one from the base branch) is what it would upload again, and
re-compressing and uploading
+ # ~7 GB costs about five minutes per run. Save on a miss, and on non-PR
runs (the canary on
+ # the default branch) so the base-branch stash stays fresh within its
retention window.
- name: "Prepare .tar file from prek cache"
run: |
- tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ if command -v pigz > /dev/null 2>&1; then
+ tar -C ~ -cf - .cache/prek | pigz > /tmp/cache-prek.tar.gz
+ else
+ tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ fi
shell: bash
- if: inputs.save-cache == 'true'
+ if: >
+ inputs.save-cache == 'true'
Review Comment:
Agreed the two-day bound is the trade-off. It is also a narrow window to get
into: the save steps run only after `prek install-hooks` succeeded, and a
tarball that fails to extract is treated as a miss and saved fresh, so a
partial stash needs a run whose install succeeded on a tree that was still
incomplete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
