This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new e098da78c501 fix(ci): make the source-release binary-file check
actually run (#19267)
e098da78c501 is described below
commit e098da78c501072e20da4cef4212ed84c3918cf6
Author: Vova Kolmakov <[email protected]>
AuthorDate: Wed Jul 15 20:04:41 2026 +0700
fix(ci): make the source-release binary-file check actually run (#19267)
* fix(ci): make the source-release binary-file check actually run
validate_source_binary_files.sh pipes every file through `file -I`, the
BSD/macOS spelling of the MIME option. GNU file rejects it, printing usage
to stderr and nothing to stdout, so the pipeline's `wc -l` always counts
zero. errexit does not catch it because the pipeline sits in a command
substitution and pipefail is not set. The check has therefore never fired on
Linux, which is every CI runner.
Ask libmagic for the charset instead of the MIME type. `file --mime` is the
long option for both GNU's `-i` and BSD's `-I`, and its charset=binary field
answers the question directly, without depending on libmagic's growing type
vocabulary: the previous MIME allowlist already yields five false positives
on master, where libmagic labels the JSON test data application/x-ndjson and
misdetects one .java source as application/javascript. Empty files, which
libmagic also reports as charset=binary, are exempted alongside the existing
release/ and src/test/ exemptions.
Add pipefail so a broken `file` invocation fails loudly rather than reading
as "no binary files found", run `file` once over a batched find/xargs
instead
of one process per path, and build the report from that single pass.
Run the check where it is meant to run. Its contract is the root of a source
release tree, which is how validate_staged_release.sh invokes it, but the
validate-source job ran it against the raw checkout, where rfc/,
docker/images/,
hudi-notebooks/ and .idea/ still exist. bot.yml now creates the source
release
directory once and runs both the binary and the copyright check inside it.
Closes #19266
* addressed review comments: make binary-file grep chain byte-safe and pin
the release/ exemption to top level
---------
Co-authored-by: Vova Kolmakov <[email protected]>
---
.github/workflows/bot.yml | 11 +++++++--
scripts/release/validate_source_binary_files.sh | 33 ++++++++++++++++++++++---
2 files changed, 38 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index 160822b26d7e..a8d0fbb47c25 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -80,11 +80,18 @@ jobs:
# wins the write. This one only runs `mvn apache-rat:check`,
finishes in well under a
# minute, and would seal a ~10 MB local repo that every build job
then has to discard and
# re-download from Maven Central.
+ # Both checks describe the source release, not the git checkout, so they
run inside the
+ # directory create_source_directory.sh produces - the same way
validate_staged_release.sh
+ # runs them inside an extracted source tarball. On the raw checkout they
would see the
+ # images under rfc/, docker/images/ and hudi-notebooks/, which the
source release excludes.
+ - name: Create source release directory
+ run: ./scripts/release/create_source_directory.sh hudi-tmp-repo
- name: Check Binary Files
- run: ./scripts/release/validate_source_binary_files.sh
+ run: |
+ cd hudi-tmp-repo
+ ./scripts/release/validate_source_binary_files.sh
- name: Check Copyright
run: |
- ./scripts/release/create_source_directory.sh hudi-tmp-repo
cd hudi-tmp-repo
./scripts/release/validate_source_copyright.sh
- name: RAT check
diff --git a/scripts/release/validate_source_binary_files.sh
b/scripts/release/validate_source_binary_files.sh
index e4ce431c71b9..80769c34d3b7 100755
--- a/scripts/release/validate_source_binary_files.sh
+++ b/scripts/release/validate_source_binary_files.sh
@@ -19,16 +19,41 @@
# under the License.
#
-# fail immediately
+# Run this from the root of a source release tree, i.e. the output of
+# create_source_directory.sh (see the validate-source job in bot.yml) or an
+# extracted source tarball (see validate_staged_release.sh). Paths that never
+# reach the source release are excluded by create_source_directory.sh, not
here.
+
+# fail immediately; pipefail so that a broken `file` invocation can never again
+# yield an empty pipeline that reads as "no binary files found"
set -o errexit
set -o nounset
+set -o pipefail
echo "Checking for binary files in the source files"
-numBinaryFiles=`find . -iname '*' | xargs -I {} file -I {} | grep -va
directory | grep -v "release/" | grep -v "/src/test/" | grep -va
'application/json' | grep -va 'text/' | grep -va 'application/xml' | grep -va
'application/json' | wc -l | sed -e s'/ //g'`
-if [ "$numBinaryFiles" -gt "0" ]; then
+# `--mime` is the long option for both GNU file's `-i` and BSD file's `-I`, so
it
+# is the only spelling that works on Linux CI and on a release manager's macOS.
+# Its charset field answers the question directly: libmagic reports
+# charset=binary for anything that is not text. Matching on the charset rather
+# than on a list of permitted MIME types keeps this robust against libmagic's
+# growing type vocabulary, which already labels the JSON test data
+# application/x-ndjson and misdetects at least one .java source as
+# application/javascript.
+mimeTypes=$(find . -type f -print0 | xargs -0 file --mime)
+
+# An empty file reports as inode/x-empty; charset=binary, and is neither
binary nor
+# a licensing concern. The top-level release/ holds the release guide's
screenshot;
+# src/test/ holds the binary fixtures (parquet, avro, hfile) that tests read.
Every
+# stage carries -a so a non-UTF-8 filename can never flip a grep into binary
mode and
+# drop a line. release/ is anchored to the top-level dir; src/test/ stays a
substring
+# because those fixtures live under every module's src/test/.
+binaryFiles=$(echo "$mimeTypes" | grep -a 'charset=binary' | grep -av
'inode/x-empty' \
+ | grep -av '^\./release/' | grep -av '/src/test/' || true)
+
+if [ -n "$binaryFiles" ]; then
echo -e "There were non-text files in source release. [ERROR]\n Please check
below\n"
- find . -iname '*' | xargs -I {} file -I {} | grep -va directory | grep -v
"release/release_guide" | grep -v "/src/test/" | grep -va 'application/json' |
grep -va 'text/' | grep -va 'application/xml'
+ echo "$binaryFiles"
exit 1
fi
echo -e "\t\tNo Binary Files in the source files? - [OK]\n"
