This is an automated email from the ASF dual-hosted git repository.
gfphoenix78 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
The following commit(s) were added to refs/heads/main by this push:
new a1e9e2d4a9 Adding additional test suite. (#735)
a1e9e2d4a9 is described below
commit a1e9e2d4a9326092985f82f79bd4155db044cfa4
Author: Ed Espino <[email protected]>
AuthorDate: Mon Dec 2 21:47:03 2024 -0800
Adding additional test suite. (#735)
Enabled the following suite:
- `ic-resgroup-v2`
Add step to display the contents of regression.diffs (if it exists)
Refactor check-skip job to handle PR-specific skip logic
- Updated GitHub Actions check-skip job:
- Modified logic to ensure skip behavior applies only to pull requests.
- Enhanced message parsing to include both PR titles and bodies.
- Ignored skip flags in direct pushes to main, ensuring full builds.
- Incorporated proper escaping of input data to prevent command injection.
This update improves the reliability and security of the check-skip
workflow, aligning it with project requirements to trigger full builds
for direct pushes while respecting skip flags in PRs.
Enhance GitHub Actions workflow with ignored test support
- Updated `build-cloudberry.yml`:
- Added parsing and reporting of ignored tests in the `test` job.
- Displayed names of ignored tests in test summaries.
- Included regression diffs display to aid debugging of test failures.
- Ensured consistent handling of failed and ignored tests across jobs.
- Updated workflow header:
- Documented ignored test handling in the test matrix.
- Highlighted regression diffs display as a debugging feature.
- Clarified reporting of failed and ignored tests in the workflow
overview.
These changes improve the visibility and debugging of test results,
ensuring ignored tests are appropriately tracked and reported without
impacting the build status.
---
.asf.yaml | 1 +
.github/workflows/build-cloudberry.yml | 2021 +++++++++++---------
.../resgroup/resgroup_memory_limit_resgroup_v2.out | 73 +
.../isolation2/isolation2_resgroup_v2_schedule | 2 +-
.../resgroup/resgroup_memory_limit_resgroup_v2.sql | 57 +
5 files changed, 1269 insertions(+), 885 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index f637517f0b..35af620149 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -91,6 +91,7 @@ github:
- ic-singlenode_regress
- ic-singlenode_isolation
- ic-singlenode_isolation2
+ - ic-resgroup-v2
- Generate Apache Cloudberry Build Report
# Pull request review requirements
diff --git a/.github/workflows/build-cloudberry.yml
b/.github/workflows/build-cloudberry.yml
index 5da4d7c97f..6ed49e76e6 100644
--- a/.github/workflows/build-cloudberry.yml
+++ b/.github/workflows/build-cloudberry.yml
@@ -17,102 +17,79 @@
# permissions and limitations under the License.
#
# --------------------------------------------------------------------
+# GitHub Actions Workflow: Apache Cloudberry Build Pipeline
+# --------------------------------------------------------------------
+# Description:
#
-# GitHub Actions Workflow: Apache Cloudberry Build
-# Description: Builds, tests, and packages Apache Cloudberry on Rocky Linux 9
-#
-# Workflow Structure:
-# 1. Skip Check:
-# - Evaluates CI skip flags in commit messages or PR titles
-# - Supports [skip ci], [ci skip], [no ci]
-# - Successfully completes all checks if skip detected
+# This workflow builds, tests, and packages Apache Cloudberry on
+# Rocky Linux 9. It ensures artifact integrity, performs installation
+# tests, validates key operations, and provides detailed test reports,
+# including handling for ignored test cases.
#
-# 2. Build Job:
-# - Sets up build environment in Rocky Linux 9 container
-# - Configures and builds Cloudberry
-# - Runs unit tests
-# - Creates RPM package and source tarball
-# - Performs build verification
-# - Uploads build artifacts
-# Key artifacts: RPM package, source tarball, build logs
+# Workflow Overview:
+# 1. **Check Skip**:
+# - Dynamically determines if the workflow should run based on CI skip
flags.
+# - Evaluates the following fields for skip flags:
+# - **Pull Request Events**: PR title and PR body.
+# - **Push Events**: Commit message of the head commit.
+# - Supports the following skip patterns (case-insensitive):
+# - `[skip ci]`
+# - `[ci skip]`
+# - `[no ci]`
+# - **Example Usage**:
+# - Add `[skip ci]` to a commit message, PR title, or body to skip the
workflow.
#
-# 3. Install Test Job:
-# - Sets up test environment in Rocky Linux 9 container
-# - Downloads build artifacts
-# - Verifies RPM package integrity
-# - Installs Cloudberry RPM
-# - Validates installation
-# Key artifacts: Installation logs, verification results
+# 2. **Build Job**:
+# - Configures and builds Apache Cloudberry.
+# - Runs unit tests and verifies build artifacts.
+# - Creates RPM packages, source tarballs, and logs.
+# - **Key Artifacts**: RPM package, source tarball, build logs.
#
-# 4. Test Job:
-# - Sets up test environment in Rocky Linux 9 container
-# - Downloads build artifacts
-# - Installs Cloudberry RPM
-# - Creates demo cluster
-# - Executes installcheck tests
-# - Parses and reports test results
-# Key artifacts: Test logs, regression files, test summaries
+# 3. **RPM Install Test Job**:
+# - Verifies RPM integrity and installs Cloudberry.
+# - Validates successful installation.
+# - **Key Artifacts**: Installation logs, verification results.
#
-# 5. Report Job:
-# - Generates comprehensive build pipeline report
-# - Aggregates status from all jobs
-# - Provides failure notifications if needed
+# 4. **Test Job (Matrix)**:
+# - Executes a test matrix to validate different scenarios.
+# - Creates a demo cluster and runs installcheck tests.
+# - Parses and reports test results, including failed and ignored tests.
+# - **Key Features**:
+# - Regression diffs are displayed if found, aiding quick debugging.
+# - Both failed and ignored test names are logged and reported.
+# - **Key Artifacts**: Test logs, regression files, test summaries.
#
-# Resource Requirements:
-# - Disk: Minimum 20GB free space
-# - Memory: Minimum 8GB RAM
-# - CPU: Recommended 4+ cores
+# 5. **Report Job**:
+# - Aggregates job results into a final report.
+# - Sends failure notifications if any step fails.
#
-# Container Images:
-# Build: apache/incubator-cloudberry:cbdb-build-rocky9-latest
-# - Rocky Linux 9 base with build dependencies
-# Test: apache/incubator-cloudberry:cbdb-test-rocky9-latest
-# - Rocky Linux 9 base with test dependencies
+# Execution Environment:
+# - **Runs On**: Ubuntu-latest with Rocky Linux 9 containers.
+# - **Resource Requirements**:
+# - Disk: Minimum 20GB free space.
+# - Memory: Minimum 8GB RAM.
+# - CPU: Recommended 4+ cores.
#
# Triggers:
-# - Push to build-devel branch
-# - Pull requests to build-devel branch
-# - Manual workflow dispatch
-#
-# Skip CI:
-# Add one of these flags to commit message or PR title:
-# - [skip ci] : Skip all CI checks
-# - [ci skip] : Skip all CI checks (alternative)
-# - [no ci] : Skip all CI checks (alternative)
+# - Push to `build-devel` branch.
+# - Pull requests to `build-devel` branch.
+# - Manual workflow dispatch.
#
-# Environment Variables:
-# LOG_RETENTION_DAYS: Number of days to retain logs (default: 7)
-# BUILD_TIMESTAMP: Build start timestamp
-# SRC_DIR: Source directory path
-# CBDB_VERSION: Cloudberry version (99.0.0)
-# BUILD_NUMBER: Build number (1)
-# GITHUB_OUTPUT: GitHub Actions output file path
-#
-# Test Matrix:
-# - ICW Optimizer=off: Basic regression tests with optimizer disabled
-# - ICW expandshrink: Cluster expansion/shrinking tests
-# # Disabled tests:
-# # - ICW Optimizer=on: Advanced tests with optimizer
-# # - ICW Parallel: Parallel execution tests
-#
-# Artifacts Generated:
-# - RPM package (1 day retention)
-# - Source tarball (1 day retention)
-# - Build and test logs (7 days retention)
-# - Test results (7 days retention)
-# - Job execution reports (7 days retention)
+# Container Images:
+# - **Build**: `apache/incubator-cloudberry:cbdb-build-rocky9-latest`
+# - **Test**: `apache/incubator-cloudberry:cbdb-test-rocky9-latest`
#
-# Dependencies:
-# - Cloudberry build scripts repository
-# - GitHub Actions artifacts for job communication
+# Artifacts:
+# - RPM Package (retention: ${{ env.LOG_RETENTION_DAYS }} days).
+# - Source Tarball (retention: ${{ env.LOG_RETENTION_DAYS }} days).
+# - Logs and Test Results (retention: ${{ env.LOG_RETENTION_DAYS }} days).
+# - Regression Diffs (retention: ${{ env.LOG_RETENTION_DAYS }} days).
#
# Notes:
-# - Requires Docker Hub authentication
-# - Uses concurrent job control
-# - Provides detailed job summaries
-# - Includes comprehensive error handling
-# - Maintains build artifacts for downstream use
-#
+# - Supports concurrent job execution.
+# - Includes robust skip logic for pull requests and pushes.
+# - Handles ignored test cases, ensuring results are comprehensive.
+# - Provides detailed logs and error handling for failed and ignored tests.
# --------------------------------------------------------------------
name: Apache Cloudberry Build
@@ -122,24 +99,49 @@ on:
branches: [main]
pull_request:
branches: [main]
- types: [opened, synchronize, reopened]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
+# Note: Step details, logs, and artifacts require users to be logged into
GitHub
+# even for public repositories. This is a GitHub security feature and cannot
+# be overridden by permissions.
+
permissions:
- contents: read
- packages: read
+ # READ permissions allow viewing repository contents
+ contents: read # Required for checking out code and reading repository
files
+
+ # READ permissions for packages (Container registry, etc)
+ packages: read # Allows reading from GitHub package registry
+
+ # WRITE permissions for actions includes read access to:
+ # - Workflow runs
+ # - Artifacts (requires GitHub login)
+ # - Logs (requires GitHub login)
actions: write
+ # READ permissions for checks API:
+ # - Step details visibility (requires GitHub login)
+ # - Check run status and details
+ checks: read
+
+ # READ permissions for pull request metadata:
+ # - PR status
+ # - Associated checks
+ # - Review states
+ pull-requests: read
+
env:
LOG_RETENTION_DAYS: 7
jobs:
- # Initial job to check for skip flags
+ ## ======================================================================
+ ## Job: check-skip
+ ## ======================================================================
+
check-skip:
runs-on: ubuntu-latest
outputs:
@@ -147,30 +149,46 @@ jobs:
steps:
- id: skip-check
shell: bash
+ env:
+ EVENT_NAME: ${{ github.event_name }}
+ PR_TITLE: ${{ github.event.pull_request.title || '' }}
+ PR_BODY: ${{ github.event.pull_request.body || '' }}
run: |
- # Get the message to check based on event type
- MESSAGE=""
- if [[ "${{ github.event_name }}" == "pull_request" ]]; then
- MESSAGE="${{ github.event.pull_request.title }}"
- else
- MESSAGE="${{ github.event.head_commit.message }}"
- fi
+ # Default to not skipping
+ echo "should_skip=false" >> "$GITHUB_OUTPUT"
- # Check for skip patterns
- if echo "$MESSAGE" | grep -qE '\[skip[ -]ci\]|\[ci[ -]skip\]|\[no[
-]ci\]'; then
- echo "should_skip=true" >> $GITHUB_OUTPUT
+ # Apply skip logic only for pull_request events
+ if [[ "$EVENT_NAME" == "pull_request" ]]; then
+ # Combine PR title and body for skip check
+ MESSAGE="${PR_TITLE}\n${PR_BODY}"
+
+ # Escape special characters using printf %s
+ ESCAPED_MESSAGE=$(printf "%s" "$MESSAGE")
+
+ echo "Checking PR title and body (escaped): $ESCAPED_MESSAGE"
+
+ # Check for skip patterns
+ if echo -e "$ESCAPED_MESSAGE" | grep -qEi '\[skip[ -]ci\]|\[ci[
-]skip\]|\[no[ -]ci\]'; then
+ echo "should_skip=true" >> "$GITHUB_OUTPUT"
+ fi
else
- echo "should_skip=false" >> $GITHUB_OUTPUT
+ echo "Skip logic is not applied for $EVENT_NAME events."
fi
- name: Report Skip Status
if: steps.skip-check.outputs.should_skip == 'true'
run: |
- echo "CI Skip flag detected - skipping all checks"
+ echo "CI Skip flag detected in PR - skipping all checks."
exit 0
+ ## ======================================================================
+ ## Job: build
+ ## ======================================================================
+
build:
name: Build Apache Cloudberry
+ env:
+ JOB_TYPE: build
needs: check-skip
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -184,273 +202,273 @@ jobs:
-h cdw
steps:
- - name: Skip Check
- if: needs.check-skip.outputs.should_skip == 'true'
- run: |
- echo "Build skipped via CI skip flag" >> $GITHUB_STEP_SUMMARY
- exit 0
-
- - name: Set build timestamp
- if: needs.check-skip.outputs.should_skip != 'true'
- id: set_timestamp # Add an ID to reference this step
- run: |
- timestamp=$(date +'%Y%m%d_%H%M%S')
- echo "timestamp=$timestamp" | tee -a $GITHUB_OUTPUT # Use
GITHUB_OUTPUT for job outputs
- echo "BUILD_TIMESTAMP=$timestamp" | tee -a $GITHUB_ENV # Also set as
environment variable
-
- - name: Checkout Apache Cloudberry (apache/cloudberry)
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/checkout@v4
- with:
- fetch-depth: 1
-
- - name: Checkout CI Build/Test Scripts (apache/cloudberry-devops-release)
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/checkout@v4
- with:
- repository: apache/cloudberry-devops-release
- path: cloudberry-devops-release
- ref: main
-
- - name: Move cloudberry-devops-release directory
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- set -eo pipefail
- if ! mv ${GITHUB_WORKSPACE}/cloudberry-devops-release
${GITHUB_WORKSPACE}/..; then
- echo "::error::Container initialization failed"
- exit 1
- fi
-
- - name: Setup Apache Cloudberry build environment
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- set -eo pipefail
- if ! su - gpadmin -c "/tmp/init_system.sh"; then
- echo "::error::Container initialization failed"
- exit 1
- fi
-
- mkdir -p build-logs/details
- chown -R gpadmin:gpadmin .
- chmod -R 755 .
- chmod 777 build-logs
-
- df -kh /
- rm -rf /__t/*
- df -kh /
-
- df -h | tee -a build-logs/details/disk-usage.log
- free -h | tee -a build-logs/details/memory-usage.log
-
- {
- echo "=== Environment Information ==="
- uname -a
- gcc --version
- df -h
- free -h
- env
- } | tee -a build-logs/details/environment.log
-
- echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a $GITHUB_ENV
-
- - name: Generate Build Job Summary Start
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- {
- echo "# Build Job Summary" >> $GITHUB_STEP_SUMMARY
- echo "## Environment" >> $GITHUB_STEP_SUMMARY
- echo "- Start Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')" >>
$GITHUB_STEP_SUMMARY
- echo "- OS Version: $(cat /etc/redhat-release)" >>
$GITHUB_STEP_SUMMARY
- echo "- GCC Version: $(gcc --version | head -n1)" >>
$GITHUB_STEP_SUMMARY
- }
-
- - name: Run Apache Cloudberry configure script
- if: needs.check-skip.outputs.should_skip != 'true'
- env:
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
- chmod +x
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/configure-cloudberry.sh
- if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/configure-cloudberry.sh";
then
- echo "::error::Configure script failed"
- exit 1
- fi
-
- - name: Run Apache Cloudberry build script
- if: needs.check-skip.outputs.should_skip != 'true'
- env:
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
- chmod +x
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/build-cloudberry.sh
- if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/build-cloudberry.sh";
then
- echo "::error::Build script failed"
- exit 1
- fi
-
- - name: Verify build artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- set -eo pipefail
-
- echo "Verifying build artifacts..."
- {
- echo "=== Build Artifacts Verification ==="
- echo "Timestamp: $(date -u)"
-
- if [ ! -d "/usr/local/cloudberry-db" ]; then
- echo "::error::Build artifacts directory not found"
+ - name: Skip Check
+ if: needs.check-skip.outputs.should_skip == 'true'
+ run: |
+ echo "Build skipped via CI skip flag" >> "$GITHUB_STEP_SUMMARY"
+ exit 0
+
+ - name: Set build timestamp
+ if: needs.check-skip.outputs.should_skip != 'true'
+ id: set_timestamp # Add an ID to reference this step
+ run: |
+ timestamp=$(date +'%Y%m%d_%H%M%S')
+ echo "timestamp=$timestamp" | tee -a "$GITHUB_OUTPUT" # Use
GITHUB_OUTPUT for job outputs
+ echo "BUILD_TIMESTAMP=$timestamp" | tee -a "$GITHUB_ENV" # Also set
as environment variable
+
+ - name: Checkout Apache Cloudberry (apache/cloudberry)
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/checkout@v4
+ with:
+ repository: apache/cloudberry
+ fetch-depth: 1
+
+ - name: Checkout CI Build/Test Scripts (apache/cloudberry-devops-release)
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/checkout@v4
+ with:
+ repository: apache/cloudberry-devops-release
+ ref: main
+ path: cloudberry-devops-release
+
+ - name: Move cloudberry-devops-release directory
+ if: needs.check-skip.outputs.should_skip != 'true'
+ run: |
+ set -eo pipefail
+ if ! mv "${GITHUB_WORKSPACE}"/cloudberry-devops-release
"${GITHUB_WORKSPACE}"/..; then
+ echo "::error::Container initialization failed"
exit 1
fi
- # Verify critical binaries
- critical_binaries=(
- "/usr/local/cloudberry-db/bin/postgres"
- "/usr/local/cloudberry-db/bin/psql"
- )
-
- echo "Checking critical binaries..."
- for binary in "${critical_binaries[@]}"; do
- if [ ! -f "$binary" ]; then
- echo "::error::Critical binary missing: $binary"
- exit 1
- fi
- if [ ! -x "$binary" ]; then
- echo "::error::Binary not executable: $binary"
- exit 1
- fi
- echo "Binary verified: $binary"
- ls -l "$binary"
- done
-
- # Test binary execution
- echo "Testing binary execution..."
- if ! /usr/local/cloudberry-db/bin/postgres --version; then
- echo "::error::postgres binary verification failed"
+ - name: Cloudberry Environment Initialization
+ if: needs.check-skip.outputs.should_skip != 'true'
+ env:
+ LOGS_DIR: build-logs
+ run: |
+ set -eo pipefail
+ if ! su - gpadmin -c "/tmp/init_system.sh"; then
+ echo "::error::Container initialization failed"
exit 1
fi
- if ! /usr/local/cloudberry-db/bin/psql --version; then
- echo "::error::psql binary verification failed"
+
+ mkdir -p "${LOGS_DIR}/details"
+ chown -R gpadmin:gpadmin .
+ chmod -R 755 .
+ chmod 777 "${LOGS_DIR}"
+
+ df -kh /
+ rm -rf /__t/*
+ df -kh /
+
+ df -h | tee -a "${LOGS_DIR}/details/disk-usage.log"
+ free -h | tee -a "${LOGS_DIR}/details/memory-usage.log"
+
+ {
+ echo "=== Environment Information ==="
+ uname -a
+ df -h
+ free -h
+ env
+ } | tee -a "${LOGS_DIR}/details/environment.log"
+
+ echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a "$GITHUB_ENV"
+
+ - name: Generate Build Job Summary Start
+ if: needs.check-skip.outputs.should_skip != 'true'
+ run: |
+ {
+ echo "# Build Job Summary"
+ echo "## Environment"
+ echo "- Start Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
+ echo "- OS Version: $(cat /etc/redhat-release)"
+ echo "- GCC Version: $(gcc --version | head -n1)"
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ - name: Run Apache Cloudberry configure script
+ if: needs.check-skip.outputs.should_skip != 'true'
+ env:
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -eo pipefail
+ chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/configure-cloudberry.sh
+ if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/configure-cloudberry.sh";
then
+ echo "::error::Configure script failed"
exit 1
fi
- echo "All build artifacts verified successfully"
- } 2>&1 | tee -a build-logs/details/build-verification.log
-
- - name: Create Source tarball, create RPM and verify artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- id: create-artifacts
- env:
- CBDB_VERSION: 99.0.0
- BUILD_NUMBER: 1
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
-
- {
- echo "=== Artifact Creation Log ==="
- echo "Timestamp: $(date -u)"
-
- # Create source tarball
- echo "Creating source tarball..."
- tar czf ${SRC_DIR}/../apache-cloudberry-incubating-src.tgz -C
${SRC_DIR}/.. ./cloudberry
- mv ${SRC_DIR}/../apache-cloudberry-incubating-src.tgz ${SRC_DIR}
-
- # Verify tarball contents
- echo "Verifying source tarball contents..."
- if ! tar tzf ${SRC_DIR}/apache-cloudberry-incubating-src.tgz >
/dev/null; then
- echo "::error::Source tarball verification failed"
+ - name: Run Apache Cloudberry build script
+ if: needs.check-skip.outputs.should_skip != 'true'
+ env:
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -eo pipefail
+ chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/build-cloudberry.sh
+ if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/build-cloudberry.sh";
then
+ echo "::error::Build script failed"
exit 1
fi
- # Create RPM
- echo "Creating RPM package..."
- rpmdev-setuptree
- ln -s
${SRC_DIR}/../cloudberry-devops-release/packaging/rpm/el/SPECS/apache-cloudberry-db-incubating.spec
${HOME}/rpmbuild/SPECS/apache-cloudberry-db-incubating.spec
- cp ${SRC_DIR}/LICENSE /usr/local/cloudberry-db
+ - name: Verify build artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ run: |
+ set -eo pipefail
- ${SRC_DIR}/../cloudberry-devops-release/scripts/build-rpm.sh
--version ${CBDB_VERSION} --release ${BUILD_NUMBER}
+ echo "Verifying build artifacts..."
+ {
+ echo "=== Build Artifacts Verification ==="
+ echo "Timestamp: $(date -u)"
- # Get OS version and move RPM
- os_version=$(grep -oP '(?<=^VERSION_ID=")[0-9]' /etc/os-release)
-
RPM_FILE=${HOME}/rpmbuild/RPMS/x86_64/apache-cloudberry-db-incubating-${CBDB_VERSION}-${BUILD_NUMBER}.el${os_version}.x86_64.rpm
- cp ${RPM_FILE} ${SRC_DIR}
+ if [ ! -d "/usr/local/cloudberry-db" ]; then
+ echo "::error::Build artifacts directory not found"
+ exit 1
+ fi
- # Get package information
- echo "Package Information:"
- rpm -qip ${RPM_FILE}
+ # Verify critical binaries
+ critical_binaries=(
+ "/usr/local/cloudberry-db/bin/postgres"
+ "/usr/local/cloudberry-db/bin/psql"
+ )
- # Verify critical files in RPM
- echo "Verifying critical files in RPM..."
- for binary in "bin/postgres" "bin/psql"; do
- if ! rpm -qlp ${RPM_FILE} | grep -q "${binary}$"; then
- echo "::error::Critical binary '${binary}' not found in RPM"
+ echo "Checking critical binaries..."
+ for binary in "${critical_binaries[@]}"; do
+ if [ ! -f "$binary" ]; then
+ echo "::error::Critical binary missing: $binary"
+ exit 1
+ fi
+ if [ ! -x "$binary" ]; then
+ echo "::error::Binary not executable: $binary"
+ exit 1
+ fi
+ echo "Binary verified: $binary"
+ ls -l "$binary"
+ done
+
+ # Test binary execution
+ echo "Testing binary execution..."
+ if ! /usr/local/cloudberry-db/bin/postgres --version; then
+ echo "::error::postgres binary verification failed"
exit 1
fi
- done
+ if ! /usr/local/cloudberry-db/bin/psql --version; then
+ echo "::error::psql binary verification failed"
+ exit 1
+ fi
+
+ echo "All build artifacts verified successfully"
+ } 2>&1 | tee -a build-logs/details/build-verification.log
+
+ - name: Create Source tarball, create RPM and verify artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ env:
+ CBDB_VERSION: 99.0.0
+ BUILD_NUMBER: 1
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -eo pipefail
+
+ {
+ echo "=== Artifact Creation Log ==="
+ echo "Timestamp: $(date -u)"
+
+ # Create source tarball
+ echo "Creating source tarball..."
+ tar czf "${SRC_DIR}"/../apache-cloudberry-incubating-src.tgz -C
"${SRC_DIR}"/.. ./cloudberry
+ mv "${SRC_DIR}"/../apache-cloudberry-incubating-src.tgz
"${SRC_DIR}"
+
+ # Verify tarball contents
+ echo "Verifying source tarball contents..."
+ if ! tar tzf "${SRC_DIR}"/apache-cloudberry-incubating-src.tgz >
/dev/null; then
+ echo "::error::Source tarball verification failed"
+ exit 1
+ fi
+
+ # Create RPM
+ echo "Creating RPM package..."
+ rpmdev-setuptree
+ ln -s
"${SRC_DIR}"/../cloudberry-devops-release/packaging/rpm/el/SPECS/apache-cloudberry-db-incubating.spec
"${HOME}"/rpmbuild/SPECS/apache-cloudberry-db-incubating.spec
+ cp "${SRC_DIR}"/LICENSE /usr/local/cloudberry-db
+
+ "${SRC_DIR}"/../cloudberry-devops-release/scripts/build-rpm.sh
--version "${CBDB_VERSION}" --release "${BUILD_NUMBER}"
+
+ # Get OS version and move RPM
+ os_version=$(grep -oP '(?<=^VERSION_ID=")[0-9]' /etc/os-release)
+
RPM_FILE="${HOME}"/rpmbuild/RPMS/x86_64/apache-cloudberry-db-incubating-"${CBDB_VERSION}"-"${BUILD_NUMBER}".el"${os_version}".x86_64.rpm
+ cp "${RPM_FILE}" "${SRC_DIR}"
+
+ # Get package information
+ echo "Package Information:"
+ rpm -qip "${RPM_FILE}"
+
+ # Verify critical files in RPM
+ echo "Verifying critical files in RPM..."
+ for binary in "bin/postgres" "bin/psql"; do
+ if ! rpm -qlp "${RPM_FILE}" | grep -q "${binary}$"; then
+ echo "::error::Critical binary '${binary}' not found in RPM"
+ exit 1
+ fi
+ done
+
+ # Record checksums
+ echo "Calculating checksums..."
+ sha256sum "${RPM_FILE}" | tee -a build-logs/details/checksums.log
+ sha256sum "${SRC_DIR}"/apache-cloudberry-incubating-src.tgz | tee
-a build-logs/details/checksums.log
+
+ echo "Artifacts created and verified successfully"
- # Record checksums
- echo "Calculating checksums..."
- sha256sum ${RPM_FILE} | tee -a build-logs/details/checksums.log
- sha256sum ${SRC_DIR}/apache-cloudberry-incubating-src.tgz | tee -a
build-logs/details/checksums.log
-
- echo "Artifacts created and verified successfully"
-
- } 2>&1 | tee -a build-logs/details/artifact-creation.log
-
- # Set outputs for subsequent steps
- echo "rpm_file=${RPM_FILE}" >> $GITHUB_OUTPUT
- echo "version=${CBDB_VERSION}" >> $GITHUB_OUTPUT
- echo "build_number=${BUILD_NUMBER}" >> $GITHUB_OUTPUT
-
- - name: Run Apache Cloudberry unittest script
- if: needs.check-skip.outputs.should_skip != 'true'
- env:
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
- chmod +x
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/unittest-cloudberry.sh
- if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/unittest-cloudberry.sh";
then
- echo "::error::Unittest script failed"
- exit 1
- fi
-
- - name: Generate Build Job Summary End
- if: always()
- run: |
- {
- echo "## Build Results" >> $GITHUB_STEP_SUMMARY
- echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')" >>
$GITHUB_STEP_SUMMARY
- }
-
- - name: Upload build logs
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/upload-artifact@v4
- with:
- name: build-logs-${{ env.BUILD_TIMESTAMP }}
- path: |
- build-logs/
- retention-days: ${{ env.LOG_RETENTION_DAYS }}
-
- - name: Upload Cloudberry RPM build artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/upload-artifact@v4
- with:
- name: apache-cloudberry-db-incubating-rpm-build-artifacts
- retention-days: ${{ env.LOG_RETENTION_DAYS }}
- if-no-files-found: error
- path: |
- *.rpm
-
- - name: Upload Cloudberry source build artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/upload-artifact@v4
- with:
- name: apache-cloudberry-db-incubating-source-build-artifacts
- retention-days: ${{ env.LOG_RETENTION_DAYS }}
- if-no-files-found: error
- path: |
- apache-cloudberry-incubating-src.tgz
+ } 2>&1 | tee -a build-logs/details/artifact-creation.log
+
+ - name: Run Apache Cloudberry unittest script
+ if: needs.check-skip.outputs.should_skip != 'true'
+ env:
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -eo pipefail
+ chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/unittest-cloudberry.sh
+ if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/unittest-cloudberry.sh";
then
+ echo "::error::Unittest script failed"
+ exit 1
+ fi
+
+ - name: Generate Build Job Summary End
+ if: always()
+ run: |
+ {
+ echo "## Build Results"
+ echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ - name: Upload build logs
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/upload-artifact@v4
+ with:
+ name: build-logs-${{ env.BUILD_TIMESTAMP }}
+ path: |
+ build-logs/
+ retention-days: ${{ env.LOG_RETENTION_DAYS }}
+
+ - name: Upload Cloudberry RPM build artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/upload-artifact@v4
+ with:
+ name: apache-cloudberry-db-incubating-rpm-build-artifacts
+ retention-days: ${{ env.LOG_RETENTION_DAYS }}
+ if-no-files-found: error
+ path: |
+ *.rpm
+
+ - name: Upload Cloudberry source build artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/upload-artifact@v4
+ with:
+ name: apache-cloudberry-db-incubating-source-build-artifacts
+ retention-days: ${{ env.LOG_RETENTION_DAYS }}
+ if-no-files-found: error
+ path: |
+ apache-cloudberry-incubating-src.tgz
+
+ ## ======================================================================
+ ## Job: rpm-install-test
+ ## ======================================================================
rpm-install-test:
name: RPM Install Test Apache Cloudberry
@@ -465,157 +483,163 @@ jobs:
-h cdw
steps:
- - name: Skip Check
- if: needs.check-skip.outputs.should_skip == 'true'
- run: |
- echo "RPM install test skipped via CI skip flag" >>
$GITHUB_STEP_SUMMARY
- exit 0
-
- - name: Download Cloudberry RPM build artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/download-artifact@v4
- with:
- name: apache-cloudberry-db-incubating-rpm-build-artifacts
- path: ${{ github.workspace }}/rpm_build_artifacts
- merge-multiple: false
-
- - name: Setup Apache Cloudberry build environment
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- set -eo pipefail
- if ! su - gpadmin -c "/tmp/init_system.sh"; then
- echo "::error::Container initialization failed"
- exit 1
- fi
-
- mkdir -p install-logs/details
- chown -R gpadmin:gpadmin .
- chmod -R 755 .
- chmod 777 install-logs
-
- df -kh /
- rm -rf /__t/*
- df -kh /
-
- df -h | tee -a install-logs/details/disk-usage.log
- free -h | tee -a install-logs/details/memory-usage.log
-
- {
- echo "=== Environment Information ==="
- uname -a
- df -h
- free -h
- env
- } | tee -a install-logs/details/environment.log
-
- echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a $GITHUB_ENV
-
- - name: Verify RPM artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- id: verify-artifacts
- run: |
- set -eo pipefail
-
- RPM_FILE=$(ls
${GITHUB_WORKSPACE}/rpm_build_artifacts/apache-cloudberry-db-incubating*.rpm)
- if [ ! -f "${RPM_FILE}" ]; then
- echo "::error::RPM file not found"
- exit 1
- fi
-
- echo "rpm_file=${RPM_FILE}" >> $GITHUB_OUTPUT
-
- echo "Verifying RPM artifacts..."
- {
- echo "=== RPM Verification Summary ==="
- echo "Timestamp: $(date -u)"
- echo "RPM File: ${RPM_FILE}"
-
- # Get RPM metadata and verify contents
- echo "Package Information:"
- rpm -qip ${RPM_FILE}
-
- # Get key RPM attributes for verification
- RPM_VERSION=$(rpm -qp --queryformat "%{VERSION}" ${RPM_FILE})
- RPM_RELEASE=$(rpm -qp --queryformat "%{RELEASE}" ${RPM_FILE})
- echo "version=${RPM_VERSION}" >> $GITHUB_OUTPUT
- echo "release=${RPM_RELEASE}" >> $GITHUB_OUTPUT
-
- # Verify expected binaries are in the RPM
- echo "Verifying critical files in RPM..."
- for binary in "bin/postgres" "bin/psql"; do
- if ! rpm -qlp ${RPM_FILE} | grep -q "${binary}$"; then
- echo "::error::Critical binary '${binary}' not found in RPM"
- exit 1
- fi
- done
+ - name: Skip Check
+ if: needs.check-skip.outputs.should_skip == 'true'
+ run: |
+ echo "RPM install test skipped via CI skip flag" >>
"$GITHUB_STEP_SUMMARY"
+ exit 0
- echo "RPM Details:"
- echo "- Version: ${RPM_VERSION}"
- echo "- Release: ${RPM_RELEASE}"
-
- # Calculate and store checksum
- echo "Checksum:"
- sha256sum ${RPM_FILE}
-
- } 2>&1 | tee -a install-logs/details/rpm-verification.log
-
- - name: Install Cloudberry RPM
- if: success() && needs.check-skip.outputs.should_skip != 'true'
- env:
- RPM_FILE: ${{ steps.verify-artifacts.outputs.rpm_file }}
- RPM_VERSION: ${{ steps.verify-artifacts.outputs.version }}
- RPM_RELEASE: ${{ steps.verify-artifacts.outputs.release }}
- run: |
- set -eo pipefail
-
- if [ -z "${RPM_FILE}" ]; then
- echo "::error::RPM_FILE environment variable is not set"
- exit 1
- fi
-
- {
- echo "=== RPM Installation Log ==="
- echo "Timestamp: $(date -u)"
- echo "RPM File: ${RPM_FILE}"
- echo "Version: ${RPM_VERSION}"
- echo "Release: ${RPM_RELEASE}"
-
- # Clean install location
- rm -rf /usr/local/cloudberry-db
-
- # Install RPM
- echo "Starting installation..."
- if ! time dnf install -y "${RPM_FILE}"; then
- echo "::error::RPM installation failed"
+ - name: Download Cloudberry RPM build artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/download-artifact@v4
+ with:
+ name: apache-cloudberry-db-incubating-rpm-build-artifacts
+ path: ${{ github.workspace }}/rpm_build_artifacts
+ merge-multiple: false
+
+ - name: Cloudberry Environment Initialization
+ if: needs.check-skip.outputs.should_skip != 'true'
+ env:
+ LOGS_DIR: install-logs
+ run: |
+ set -eo pipefail
+ if ! su - gpadmin -c "/tmp/init_system.sh"; then
+ echo "::error::Container initialization failed"
exit 1
fi
- echo "Installation completed successfully"
- rpm -qi apache-cloudberry-db-incubating
- echo "Installed files:"
- rpm -ql apache-cloudberry-db-incubating
- } 2>&1 | tee -a install-logs/details/rpm-installation.log
-
- - name: Upload install logs
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/upload-artifact@v4
- with:
- name: install-logs-${{ needs.build.outputs.build_timestamp }}
- path: |
- install-logs/
- retention-days: ${{ env.LOG_RETENTION_DAYS }}
-
- - name: Generate Install Test Job Summary End
- if: always()
- shell: bash {0}
- run: |
- {
- echo "# Installed Package Summary"
- echo "\`\`\`"
-
- rpm -qi apache-cloudberry-db-incubating
- echo "\`\`\`"
- } >> $GITHUB_STEP_SUMMARY || true
+ mkdir -p "${LOGS_DIR}/details"
+ chown -R gpadmin:gpadmin .
+ chmod -R 755 .
+ chmod 777 "${LOGS_DIR}"
+
+ df -kh /
+ rm -rf /__t/*
+ df -kh /
+
+ df -h | tee -a "${LOGS_DIR}/details/disk-usage.log"
+ free -h | tee -a "${LOGS_DIR}/details/memory-usage.log"
+
+ {
+ echo "=== Environment Information ==="
+ uname -a
+ df -h
+ free -h
+ env
+ } | tee -a "${LOGS_DIR}/details/environment.log"
+
+ echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a "$GITHUB_ENV"
+
+ - name: Verify RPM artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ id: verify-artifacts
+ run: |
+ set -eo pipefail
+
+ RPM_FILE=$(ls
"${GITHUB_WORKSPACE}"/rpm_build_artifacts/apache-cloudberry-db-incubating*.rpm)
+ if [ ! -f "${RPM_FILE}" ]; then
+ echo "::error::RPM file not found"
+ exit 1
+ fi
+
+ echo "rpm_file=${RPM_FILE}" >> "$GITHUB_OUTPUT"
+
+ echo "Verifying RPM artifacts..."
+ {
+ echo "=== RPM Verification Summary ==="
+ echo "Timestamp: $(date -u)"
+ echo "RPM File: ${RPM_FILE}"
+
+ # Get RPM metadata and verify contents
+ echo "Package Information:"
+ rpm -qip "${RPM_FILE}"
+
+ # Get key RPM attributes for verification
+ RPM_VERSION=$(rpm -qp --queryformat "%{VERSION}" "${RPM_FILE}")
+ RPM_RELEASE=$(rpm -qp --queryformat "%{RELEASE}" "${RPM_FILE}")
+ echo "version=${RPM_VERSION}" >> "$GITHUB_OUTPUT"
+ echo "release=${RPM_RELEASE}" >> "$GITHUB_OUTPUT"
+
+ # Verify expected binaries are in the RPM
+ echo "Verifying critical files in RPM..."
+ for binary in "bin/postgres" "bin/psql"; do
+ if ! rpm -qlp "${RPM_FILE}" | grep -q "${binary}$"; then
+ echo "::error::Critical binary '${binary}' not found in RPM"
+ exit 1
+ fi
+ done
+
+ echo "RPM Details:"
+ echo "- Version: ${RPM_VERSION}"
+ echo "- Release: ${RPM_RELEASE}"
+
+ # Calculate and store checksum
+ echo "Checksum:"
+ sha256sum "${RPM_FILE}"
+
+ } 2>&1 | tee -a install-logs/details/rpm-verification.log
+
+ - name: Install Cloudberry RPM
+ if: success() && needs.check-skip.outputs.should_skip != 'true'
+ env:
+ RPM_FILE: ${{ steps.verify-artifacts.outputs.rpm_file }}
+ RPM_VERSION: ${{ steps.verify-artifacts.outputs.version }}
+ RPM_RELEASE: ${{ steps.verify-artifacts.outputs.release }}
+ run: |
+ set -eo pipefail
+
+ if [ -z "${RPM_FILE}" ]; then
+ echo "::error::RPM_FILE environment variable is not set"
+ exit 1
+ fi
+
+ {
+ echo "=== RPM Installation Log ==="
+ echo "Timestamp: $(date -u)"
+ echo "RPM File: ${RPM_FILE}"
+ echo "Version: ${RPM_VERSION}"
+ echo "Release: ${RPM_RELEASE}"
+
+ # Clean install location
+ rm -rf /usr/local/cloudberry-db
+
+ # Install RPM
+ echo "Starting installation..."
+ if ! time dnf install -y "${RPM_FILE}"; then
+ echo "::error::RPM installation failed"
+ exit 1
+ fi
+
+ echo "Installation completed successfully"
+ rpm -qi apache-cloudberry-db-incubating
+ echo "Installed files:"
+ rpm -ql apache-cloudberry-db-incubating
+ } 2>&1 | tee -a install-logs/details/rpm-installation.log
+
+ - name: Upload install logs
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/upload-artifact@v4
+ with:
+ name: install-logs-${{ needs.build.outputs.build_timestamp }}
+ path: |
+ install-logs/
+ retention-days: ${{ env.LOG_RETENTION_DAYS }}
+
+ - name: Generate Install Test Job Summary End
+ if: always()
+ shell: bash {0}
+ run: |
+ {
+ echo "# Installed Package Summary"
+ echo "\`\`\`"
+
+ rpm -qi apache-cloudberry-db-incubating
+ echo "\`\`\`"
+ } >> "$GITHUB_STEP_SUMMARY" || true
+
+ ## ======================================================================
+ ## Job: test
+ ## ======================================================================
test:
name: ${{ matrix.test }}
@@ -631,6 +655,7 @@ jobs:
- ic-singlenode_regress
- ic-singlenode_isolation
- ic-singlenode_isolation2
+ - ic-resgroup-v2
include:
- test: ic-good-opt-off
make_target: installcheck-good
@@ -654,414 +679,642 @@ jobs:
make_target: installcheck-singlenode
make_directory: --directory=src/test/singlenode_isolation2
num_primary_mirror_pairs: 0
+ - test: ic-resgroup-v2
+ make_target: installcheck-resgroup-v2
+ make_directory: --directory=src/test/isolation2
+ num_primary_mirror_pairs: 3
container:
image: apache/incubator-cloudberry:cbdb-build-rocky9-latest
options: >-
+ --privileged
--user root
- -h cdw
+ --hostname cdw
--shm-size=2gb
+ --cgroupns=host
+ -v /sys/fs/cgroup:/sys/fs/cgroup:rw
steps:
+ - name: Skip Check
+ if: needs.check-skip.outputs.should_skip == 'true'
+ run: |
+ echo "Test ${{ matrix.test }} skipped via CI skip flag" >>
"$GITHUB_STEP_SUMMARY"
+ exit 0
- - name: Skip Check
- if: needs.check-skip.outputs.should_skip == 'true'
- run: |
- echo "Test ${{ matrix.test }} skipped via CI skip flag" >>
$GITHUB_STEP_SUMMARY
- exit 0
-
- - name: Use timestamp from previous job
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- echo "Timestamp from output: ${{ needs.build.outputs.build_timestamp
}}"
-
- - name: Checkout CI Build/Test Scripts (apache/cloudberry-devops-release)
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/checkout@v4
- with:
- repository: apache/cloudberry-devops-release
- path: cloudberry-devops-release
- ref: main
-
- - name: Move cloudberry-devops-release directory
- if: needs.check-skip.outputs.should_skip != 'true'
- run: |
- set -eo pipefail
- if ! mv ${GITHUB_WORKSPACE}/cloudberry-devops-release
${GITHUB_WORKSPACE}/..; then
- echo "::error::Container initialization failed"
- exit 1
- fi
-
- - name: Setup Apache Cloudberry build environment
- run: |
- set -eo pipefail
- if ! su - gpadmin -c "/tmp/init_system.sh"; then
- echo "::error::Container initialization failed"
- exit 1
- fi
-
- mkdir -p build-logs/details
- chown -R gpadmin:gpadmin .
- chmod -R 755 .
- chmod 777 build-logs
-
- df -kh /
- rm -rf /__t/*
- df -kh /
-
- df -h | tee -a build-logs/details/disk-usage.log
- free -h | tee -a build-logs/details/memory-usage.log
-
- {
- echo "=== Environment Information ==="
- uname -a
- df -h
- free -h
- env
- } | tee -a build-logs/details/environment.log
-
- echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a $GITHUB_ENV
-
- - name: "Generate Test Job Summary Start: ${{ matrix.test }}"
- if: always()
- run: |
- {
- echo "# Test Job Summary: ${{ matrix.test }}" >> $GITHUB_STEP_SUMMARY
- echo "## Environment" >> $GITHUB_STEP_SUMMARY
- echo "- Start Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')" >>
$GITHUB_STEP_SUMMARY
-
- if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
- echo "## Skip Status" >> $GITHUB_STEP_SUMMARY
- echo "✓ Test execution skipped via CI skip flag" >>
$GITHUB_STEP_SUMMARY
- else
- echo "- OS Version: $(cat /etc/redhat-release)" >>
$GITHUB_STEP_SUMMARY
+ - name: Use timestamp from previous job
+ if: needs.check-skip.outputs.should_skip != 'true'
+ run: |
+ echo "Timestamp from output: ${{ needs.build.outputs.build_timestamp
}}"
+
+ - name: Checkout CI Build/Test Scripts (apache/cloudberry-devops-release)
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/checkout@v4
+ with:
+ repository: apache/cloudberry-devops-release
+ ref: main
+ path: cloudberry-devops-release
+
+ - name: Move cloudberry-devops-release directory
+ if: needs.check-skip.outputs.should_skip != 'true'
+ run: |
+ set -eo pipefail
+ if ! mv "${GITHUB_WORKSPACE}"/cloudberry-devops-release
"${GITHUB_WORKSPACE}"/..; then
+ echo "::error::Container initialization failed"
+ exit 1
fi
- }
-
- - name: Download Cloudberry RPM build artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/download-artifact@v4
- with:
- name: apache-cloudberry-db-incubating-rpm-build-artifacts
- path: ${{ github.workspace }}/rpm_build_artifacts
- merge-multiple: false
-
- - name: Download Cloudberry Source build artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- uses: actions/download-artifact@v4
- with:
- name: apache-cloudberry-db-incubating-source-build-artifacts
- path: ${{ github.workspace }}/source_build_artifacts
- merge-multiple: false
-
- - name: Verify downloaded artifacts
- if: needs.check-skip.outputs.should_skip != 'true'
- id: verify-artifacts
- run: |
- set -eo pipefail
-
- SRC_TARBALL_FILE=$(ls
${GITHUB_WORKSPACE}/source_build_artifacts/apache-cloudberry-incubating-src.tgz)
- if [ ! -f "${SRC_TARBALL_FILE}" ]; then
- echo "::error::SRC TARBALL file not found"
- exit 1
- fi
-
- echo "src_tarball_file=${SRC_TARBALL_FILE}" >> $GITHUB_OUTPUT
-
- echo "Verifying SRC TARBALL artifacts..."
- {
- echo "=== SRC TARBALL Verification Summary ==="
- echo "Timestamp: $(date -u)"
- echo "SRC TARBALL File: ${SRC_TARBALL_FILE}"
-
- # Calculate and store checksum
- echo "Checksum:"
- sha256sum ${SRC_TARBALL_FILE}
-
- } 2>&1 | tee -a build-logs/details/src-tarball-verification.log
-
- RPM_FILE=$(ls
${GITHUB_WORKSPACE}/rpm_build_artifacts/apache-cloudberry-db-incubating*.rpm)
- if [ ! -f "${RPM_FILE}" ]; then
- echo "::error::RPM file not found"
- exit 1
- fi
-
- echo "rpm_file=${RPM_FILE}" >> $GITHUB_OUTPUT
-
- echo "Verifying RPM artifacts..."
- {
- echo "=== RPM Verification Summary ==="
- echo "Timestamp: $(date -u)"
- echo "RPM File: ${RPM_FILE}"
-
- # Get RPM metadata and verify contents
- echo "Package Information:"
- rpm -qip ${RPM_FILE}
-
- # Get key RPM attributes for verification
- RPM_VERSION=$(rpm -qp --queryformat "%{VERSION}" ${RPM_FILE})
- RPM_RELEASE=$(rpm -qp --queryformat "%{RELEASE}" ${RPM_FILE})
- echo "version=${RPM_VERSION}" >> $GITHUB_OUTPUT
- echo "release=${RPM_RELEASE}" >> $GITHUB_OUTPUT
-
- # Verify expected binaries are in the RPM
- echo "Verifying critical files in RPM..."
- for binary in "bin/postgres" "bin/psql"; do
- if ! rpm -qlp ${RPM_FILE} | grep -q "${binary}$"; then
- echo "::error::Critical binary '${binary}' not found in RPM"
- exit 1
+
+ - name: Cloudberry Environment Initialization
+ env:
+ LOGS_DIR: build-logs
+ run: |
+ set -eo pipefail
+ if ! su - gpadmin -c "/tmp/init_system.sh"; then
+ echo "::error::Container initialization failed"
+ exit 1
+ fi
+
+ mkdir -p "${LOGS_DIR}/details"
+ chown -R gpadmin:gpadmin .
+ chmod -R 755 .
+ chmod 777 "${LOGS_DIR}"
+
+ df -kh /
+ rm -rf /__t/*
+ df -kh /
+
+ df -h | tee -a "${LOGS_DIR}/details/disk-usage.log"
+ free -h | tee -a "${LOGS_DIR}/details/memory-usage.log"
+
+ {
+ echo "=== Environment Information ==="
+ uname -a
+ df -h
+ free -h
+ env
+ } | tee -a "${LOGS_DIR}/details/environment.log"
+
+ echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a "$GITHUB_ENV"
+
+ - name: Setup cgroups
+ if: needs.check-skip.outputs.should_skip != 'true'
+ shell: bash
+ run: |
+ set -uxo pipefail
+
+ echo "Current mounts:"
+ mount | grep cgroup
+
+ CGROUP_BASEDIR=/sys/fs/cgroup
+
+ # 1. Basic setup with permissions
+ sudo chmod -R 777 ${CGROUP_BASEDIR}/
+ sudo mkdir -p ${CGROUP_BASEDIR}/gpdb
+ sudo chmod -R 777 ${CGROUP_BASEDIR}/gpdb
+ sudo chown -R gpadmin:gpadmin ${CGROUP_BASEDIR}/gpdb
+
+ # 2. Enable controllers
+ sudo bash -c "echo '+cpu +cpuset +memory +io' >
${CGROUP_BASEDIR}/cgroup.subtree_control" || true
+ sudo bash -c "echo '+cpu +cpuset +memory +io' >
${CGROUP_BASEDIR}/gpdb/cgroup.subtree_control" || true
+
+ # 3. CPU settings
+ sudo bash -c "echo 'max 100000' > ${CGROUP_BASEDIR}/gpdb/cpu.max" ||
true
+ sudo bash -c "echo '100' > ${CGROUP_BASEDIR}/gpdb/cpu.weight" || true
+ sudo bash -c "echo '0' > ${CGROUP_BASEDIR}/gpdb/cpu.weight.nice" ||
true
+ sudo bash -c "echo 0-$(( $(nproc) - 1 )) >
${CGROUP_BASEDIR}/gpdb/cpuset.cpus" || true
+ sudo bash -c "echo '0' > ${CGROUP_BASEDIR}/gpdb/cpuset.mems" || true
+
+ # 4. Memory settings
+ sudo bash -c "echo 'max' > ${CGROUP_BASEDIR}/gpdb/memory.max" || true
+ sudo bash -c "echo '0' > ${CGROUP_BASEDIR}/gpdb/memory.min" || true
+ sudo bash -c "echo 'max' > ${CGROUP_BASEDIR}/gpdb/memory.high" ||
true
+
+ # 5. IO settings
+ echo "Available block devices:"
+ lsblk
+
+ sudo bash -c "
+ if [ -f \${CGROUP_BASEDIR}/gpdb/io.stat ]; then
+ echo 'Detected IO devices:'
+ cat \${CGROUP_BASEDIR}/gpdb/io.stat
fi
+ echo '' > \${CGROUP_BASEDIR}/gpdb/io.max || true
+ "
+
+ # 6. Fix permissions again after all writes
+ sudo chmod -R 777 ${CGROUP_BASEDIR}/gpdb
+ sudo chown -R gpadmin:gpadmin ${CGROUP_BASEDIR}/gpdb
+
+ # 7. Check required files
+ echo "Checking required files:"
+ required_files=(
+ "cgroup.procs"
+ "cpu.max"
+ "cpu.pressure"
+ "cpu.weight"
+ "cpu.weight.nice"
+ "cpu.stat"
+ "cpuset.cpus"
+ "cpuset.mems"
+ "cpuset.cpus.effective"
+ "cpuset.mems.effective"
+ "memory.current"
+ "io.max"
+ )
+
+ for file in "${required_files[@]}"; do
+ if [ -f "${CGROUP_BASEDIR}/gpdb/$file" ]; then
+ echo "✓ $file exists"
+ ls -l "${CGROUP_BASEDIR}/gpdb/$file"
+ else
+ echo "✗ $file missing"
+ fi
done
- echo "RPM Details:"
- echo "- Version: ${RPM_VERSION}"
- echo "- Release: ${RPM_RELEASE}"
-
- # Calculate and store checksum
- echo "Checksum:"
- sha256sum ${RPM_FILE}
-
- } 2>&1 | tee -a build-logs/details/rpm-verification.log
-
- - name: Install Cloudberry RPM
- if: success() && needs.check-skip.outputs.should_skip != 'true'
- env:
- RPM_FILE: ${{ steps.verify-artifacts.outputs.rpm_file }}
- RPM_VERSION: ${{ steps.verify-artifacts.outputs.version }}
- RPM_RELEASE: ${{ steps.verify-artifacts.outputs.release }}
- run: |
- set -eo pipefail
-
- if [ -z "${RPM_FILE}" ]; then
- echo "::error::RPM_FILE environment variable is not set"
- exit 1
- fi
-
- {
- echo "=== RPM Installation Log ==="
- echo "Timestamp: $(date -u)"
- echo "RPM File: ${RPM_FILE}"
- echo "Version: ${RPM_VERSION}"
- echo "Release: ${RPM_RELEASE}"
-
- # Clean install location
- rm -rf /usr/local/cloudberry-db
-
- # Install RPM
- echo "Starting installation..."
- if ! time dnf install -y "${RPM_FILE}"; then
- echo "::error::RPM installation failed"
+ # 8. Test subdirectory creation
+ echo "Testing subdirectory creation..."
+ sudo -u gpadmin bash -c "
+ TEST_DIR=\${CGROUP_BASEDIR}/gpdb/test6448
+ if mkdir -p \$TEST_DIR; then
+ echo 'Created test directory'
+ sudo chmod -R 777 \$TEST_DIR
+ if echo \$\$ > \$TEST_DIR/cgroup.procs; then
+ echo 'Successfully wrote to cgroup.procs'
+ cat \$TEST_DIR/cgroup.procs
+ # Move processes back to parent before cleanup
+ echo \$\$ > \${CGROUP_BASEDIR}/gpdb/cgroup.procs
+ else
+ echo 'Failed to write to cgroup.procs'
+ ls -la \$TEST_DIR/cgroup.procs
+ fi
+ ls -la \$TEST_DIR/
+ rmdir \$TEST_DIR || {
+ echo 'Moving all processes to parent before cleanup'
+ cat \$TEST_DIR/cgroup.procs | while read pid; do
+ echo \$pid > \${CGROUP_BASEDIR}/gpdb/cgroup.procs
2>/dev/null || true
+ done
+ rmdir \$TEST_DIR
+ }
+ else
+ echo 'Failed to create test directory'
+ fi
+ "
+
+ # 9. Verify setup as gpadmin user
+ echo "Testing cgroup access as gpadmin..."
+ sudo -u gpadmin bash -c "
+ echo 'Checking mounts...'
+ mount | grep cgroup
+
+ echo 'Checking /proc/self/mounts...'
+ cat /proc/self/mounts | grep cgroup
+
+ if ! grep -q cgroup2 /proc/self/mounts; then
+ echo 'ERROR: cgroup2 mount NOT visible to gpadmin'
+ exit 1
+ fi
+ echo 'SUCCESS: cgroup2 mount visible to gpadmin'
+
+ if ! [ -w ${CGROUP_BASEDIR}/gpdb ]; then
+ echo 'ERROR: gpadmin cannot write to gpdb cgroup'
+ exit 1
+ fi
+ echo 'SUCCESS: gpadmin can write to gpdb cgroup'
+
+ echo 'Verifying key files content:'
+ echo 'cpu.max:'
+ cat ${CGROUP_BASEDIR}/gpdb/cpu.max || echo 'Failed to read cpu.max'
+ echo 'cpuset.cpus:'
+ cat ${CGROUP_BASEDIR}/gpdb/cpuset.cpus || echo 'Failed to read
cpuset.cpus'
+ echo 'cgroup.subtree_control:'
+ cat ${CGROUP_BASEDIR}/gpdb/cgroup.subtree_control || echo 'Failed
to read cgroup.subtree_control'
+ "
+
+ # 10. Show final state
+ echo "Final cgroup state:"
+ ls -la ${CGROUP_BASEDIR}/gpdb/
+
+ echo "Cgroup setup completed successfully"
+
+ - name: "Generate Test Job Summary Start: ${{ matrix.test }}"
+ if: always()
+ run: |
+ {
+ echo "# Test Job Summary: ${{ matrix.test }}"
+ echo "## Environment"
+ echo "- Start Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
+
+ if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
+ echo "## Skip Status"
+ echo "✓ Test execution skipped via CI skip flag"
+ else
+ echo "- OS Version: $(cat /etc/redhat-release)"
+ fi
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ - name: Download Cloudberry RPM build artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/download-artifact@v4
+ with:
+ name: apache-cloudberry-db-incubating-rpm-build-artifacts
+ path: ${{ github.workspace }}/rpm_build_artifacts
+ merge-multiple: false
+
+ - name: Download Cloudberry Source build artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ uses: actions/download-artifact@v4
+ with:
+ name: apache-cloudberry-db-incubating-source-build-artifacts
+ path: ${{ github.workspace }}/source_build_artifacts
+ merge-multiple: false
+
+ - name: Verify downloaded artifacts
+ if: needs.check-skip.outputs.should_skip != 'true'
+ id: verify-artifacts
+ run: |
+ set -eo pipefail
+
+ SRC_TARBALL_FILE=$(ls
"${GITHUB_WORKSPACE}"/source_build_artifacts/apache-cloudberry-incubating-src.tgz)
+ if [ ! -f "${SRC_TARBALL_FILE}" ]; then
+ echo "::error::SRC TARBALL file not found"
exit 1
fi
- echo "Installation completed successfully"
- rpm -qi apache-cloudberry-db-incubating
- } 2>&1 | tee -a build-logs/details/rpm-installation.log
-
- - name: Extract source tarball
- if: success() && needs.check-skip.outputs.should_skip != 'true'
- env:
- SRC_TARBALL_FILE: ${{ steps.verify-artifacts.outputs.src_tarball_file
}}
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
-
- {
- echo "=== Source Extraction Log ==="
- echo "Timestamp: $(date -u)"
-
- echo "Starting extraction..."
- if ! time tar zxf ${SRC_TARBALL_FILE} -C ${SRC_DIR}/.. ; then
- echo "::error::Source extraction failed"
+ echo "src_tarball_file=${SRC_TARBALL_FILE}" >> "$GITHUB_OUTPUT"
+
+ echo "Verifying SRC TARBALL artifacts..."
+ {
+ echo "=== SRC TARBALL Verification Summary ==="
+ echo "Timestamp: $(date -u)"
+ echo "SRC TARBALL File: ${SRC_TARBALL_FILE}"
+
+ # Calculate and store checksum
+ echo "Checksum:"
+ sha256sum "${SRC_TARBALL_FILE}"
+
+ } 2>&1 | tee -a build-logs/details/src-tarball-verification.log
+
+ RPM_FILE=$(ls
"${GITHUB_WORKSPACE}"/rpm_build_artifacts/apache-cloudberry-db-incubating*.rpm)
+ if [ ! -f "${RPM_FILE}" ]; then
+ echo "::error::RPM file not found"
exit 1
fi
- echo "Extraction completed successfully"
- echo "Extracted contents:"
- ls -la "${SRC_DIR}/../cloudberry"
- echo "Directory size:"
- du -sh "${SRC_DIR}/../cloudberry"
- } 2>&1 | tee -a build-logs/details/source-extraction.log
-
- - name: Create Apache Cloudberry demo cluster
- if: success() && needs.check-skip.outputs.should_skip != 'true'
- env:
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
-
- {
- chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh
- if ! time su - gpadmin -c "cd ${SRC_DIR} &&
NUM_PRIMARY_MIRROR_PAIRS='${{ matrix.num_primary_mirror_pairs }}'
SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh";
then
- echo "::error::Demo cluster creation failed"
+ echo "rpm_file=${RPM_FILE}" >> "$GITHUB_OUTPUT"
+
+ echo "Verifying RPM artifacts..."
+ {
+ echo "=== RPM Verification Summary ==="
+ echo "Timestamp: $(date -u)"
+ echo "RPM File: ${RPM_FILE}"
+
+ # Get RPM metadata and verify contents
+ echo "Package Information:"
+ rpm -qip "${RPM_FILE}"
+
+ # Get key RPM attributes for verification
+ RPM_VERSION=$(rpm -qp --queryformat "%{VERSION}" "${RPM_FILE}")
+ RPM_RELEASE=$(rpm -qp --queryformat "%{RELEASE}" "${RPM_FILE}")
+ echo "version=${RPM_VERSION}" >> "$GITHUB_OUTPUT"
+ echo "release=${RPM_RELEASE}" >> "$GITHUB_OUTPUT"
+
+ # Verify expected binaries are in the RPM
+ echo "Verifying critical files in RPM..."
+ for binary in "bin/postgres" "bin/psql"; do
+ if ! rpm -qlp "${RPM_FILE}" | grep -q "${binary}$"; then
+ echo "::error::Critical binary '${binary}' not found in RPM"
+ exit 1
+ fi
+ done
+
+ echo "RPM Details:"
+ echo "- Version: ${RPM_VERSION}"
+ echo "- Release: ${RPM_RELEASE}"
+
+ # Calculate and store checksum
+ echo "Checksum:"
+ sha256sum "${RPM_FILE}"
+
+ } 2>&1 | tee -a build-logs/details/rpm-verification.log
+
+ - name: Install Cloudberry RPM
+ if: success() && needs.check-skip.outputs.should_skip != 'true'
+ env:
+ RPM_FILE: ${{ steps.verify-artifacts.outputs.rpm_file }}
+ RPM_VERSION: ${{ steps.verify-artifacts.outputs.version }}
+ RPM_RELEASE: ${{ steps.verify-artifacts.outputs.release }}
+ run: |
+ set -eo pipefail
+
+ if [ -z "${RPM_FILE}" ]; then
+ echo "::error::RPM_FILE environment variable is not set"
exit 1
fi
- } 2>&1 | tee -a build-logs/details/create-cloudberry-demo-cluster.log
-
- - name: "Run Tests: ${{ matrix.test }}"
- if: success() && needs.check-skip.outputs.should_skip != 'true'
- env:
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -eo pipefail
- {
-
- # Construct PGOPTIONS from pg_settings
- PG_OPTS=""
- if [[ -n "${{ matrix.pg_settings.optimizer }}" ]]; then
- PG_OPTS="$PG_OPTS -c optimizer=${{ matrix.pg_settings.optimizer }}"
- fi
- chmod +x
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/test-cloudberry.sh
- if ! time su - gpadmin -c "cd ${SRC_DIR} && \
+ {
+ echo "=== RPM Installation Log ==="
+ echo "Timestamp: $(date -u)"
+ echo "RPM File: ${RPM_FILE}"
+ echo "Version: ${RPM_VERSION}"
+ echo "Release: ${RPM_RELEASE}"
+
+ # Clean install location
+ rm -rf /usr/local/cloudberry-db
+
+ # Install RPM
+ echo "Starting installation..."
+ if ! time dnf install -y "${RPM_FILE}"; then
+ echo "::error::RPM installation failed"
+ exit 1
+ fi
+
+ echo "Installation completed successfully"
+ rpm -qi apache-cloudberry-db-incubating
+ } 2>&1 | tee -a build-logs/details/rpm-installation.log
+
+ - name: Extract source tarball
+ if: success() && needs.check-skip.outputs.should_skip != 'true'
+ env:
+ SRC_TARBALL_FILE: ${{
steps.verify-artifacts.outputs.src_tarball_file }}
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -eo pipefail
+
+ {
+ echo "=== Source Extraction Log ==="
+ echo "Timestamp: $(date -u)"
+
+ echo "Starting extraction..."
+ if ! time tar zxf "${SRC_TARBALL_FILE}" -C "${SRC_DIR}"/.. ; then
+ echo "::error::Source extraction failed"
+ exit 1
+ fi
+
+ echo "Extraction completed successfully"
+ echo "Extracted contents:"
+ ls -la "${SRC_DIR}/../cloudberry"
+ echo "Directory size:"
+ du -sh "${SRC_DIR}/../cloudberry"
+ } 2>&1 | tee -a build-logs/details/source-extraction.log
+
+ - name: Create Apache Cloudberry demo cluster
+ if: success() && needs.check-skip.outputs.should_skip != 'true'
+ env:
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -eo pipefail
+
+ {
+ chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh
+ if ! time su - gpadmin -c "cd ${SRC_DIR} &&
NUM_PRIMARY_MIRROR_PAIRS='${{ matrix.num_primary_mirror_pairs }}'
SRC_DIR=${SRC_DIR}
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh";
then
+ echo "::error::Demo cluster creation failed"
+ exit 1
+ fi
+
+ } 2>&1 | tee -a build-logs/details/create-cloudberry-demo-cluster.log
+
+ - name: "Run Tests: ${{ matrix.test }}"
+ if: success() && needs.check-skip.outputs.should_skip != 'true'
+ env:
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -o pipefail
+ {
+
+ # Construct PGOPTIONS from pg_settings
+ PG_OPTS=""
+ if [[ -n "${{ matrix.pg_settings.optimizer }}" ]]; then
+ PG_OPTS="$PG_OPTS -c optimizer=${{ matrix.pg_settings.optimizer
}}"
+ fi
+
+ chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/test-cloudberry.sh
+ if ! time su - gpadmin -c "cd ${SRC_DIR} && \
+ MAKE_NAME='${{ matrix.test }}' \
+ MAKE_TARGET='${{ matrix.make_target }}' \
+ MAKE_DIRECTORY='${{ matrix.make_directory }}' \
+ PGOPTIONS='${PG_OPTS}' \
+ SRC_DIR='${SRC_DIR}' \
+
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/test-cloudberry.sh";
then
+ echo "::error::Test execution failed"
+ df -kh /
+ exit 1
+ fi
+ } 2>&1 | tee -a build-logs/details/make-${{ matrix.test }}.log
+
+ - name: Parse Test Results
+ id: test-results
+ if: always()
+ shell: bash {0}
+ env:
+ SRC_DIR: ${{ github.workspace }}
+ run: |
+ set -o pipefail
+ {
+ # Check if the workflow was cancelled
+ if [[ "${{ job.status }}" == "cancelled" ]]; then
+ echo "Workflow was cancelled - recording cancelled status"
+ {
+ echo "status=cancelled"
+ echo "total_tests=0"
+ echo "failed_tests=0"
+ echo "passed_tests=0"
+ echo "ignored_tests=0"
+ } >> "$GITHUB_OUTPUT"
+ echo "Cancellation status recorded" | tee -a
build-logs/details/parse-test-results-${{ matrix.test }}.log
+ exit 0
+ fi
+
+ # Check for skipped tests
+ if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
+ echo "Tests skipped - setting default values"
+ {
+ echo "status=skipped"
+ echo "total_tests=0"
+ echo "failed_tests=0"
+ echo "passed_tests=0"
+ echo "ignored_tests=0"
+ } >> "$GITHUB_OUTPUT"
+ echo "Skip status recorded" | tee -a
build-logs/details/parse-test-results-${{ matrix.test }}.log
+ exit 0
+ fi
+
+ # Ensure log file exists and has content
+ LOG_FILE="build-logs/details/make-${{ matrix.test }}.log"
+ if [[ ! -f "$LOG_FILE" ]]; then
+ echo "::warning::Test log file not found, using default values"
+ {
+ echo "status=no_log"
+ echo "total_tests=0"
+ echo "failed_tests=0"
+ echo "passed_tests=0"
+ echo "ignored_tests=0"
+ } >> "$GITHUB_OUTPUT"
+ exit 0
+ fi
+
+ # Check if log file is empty or contains only whitespace
+ if [[ ! -s "$LOG_FILE" ]] || ! grep -q '[^[:space:]]' "$LOG_FILE";
then
+ echo "::warning::Test log file is empty, using default values"
+ {
+ echo "status=empty_log"
+ echo "total_tests=0"
+ echo "failed_tests=0"
+ echo "passed_tests=0"
+ echo "ignored_tests=0"
+ } >> "$GITHUB_OUTPUT"
+ exit 0
+ fi
+
+ # Attempt to parse results
+ chmod +x
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/parse-test-results.sh
MAKE_NAME='${{ matrix.test }}' \
- MAKE_TARGET='${{ matrix.make_target }}' \
- MAKE_DIRECTORY='${{ matrix.make_directory }}' \
- PGOPTIONS='${PG_OPTS}' \
- SRC_DIR='${SRC_DIR}' \
-
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/test-cloudberry.sh";
then
- echo "::error::Test execution failed"
- df -kh /
- exit 1
- fi
- } 2>&1 | tee -a build-logs/details/make-${{ matrix.test }}.log
-
- - name: Parse Test Results
- id: test-results
- if: always()
- shell: bash {0}
- env:
- SRC_DIR: ${{ github.workspace }}
- run: |
- set -o pipefail
- {
- if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
- echo "Tests skipped - setting default values"
- echo "status=skipped" >> $GITHUB_OUTPUT
- echo "total_tests=0" >> $GITHUB_OUTPUT
- echo "failed_tests=0" >> $GITHUB_OUTPUT
- echo "passed_tests=0" >> $GITHUB_OUTPUT
- echo "Skip status recorded" | tee -a
build-logs/details/parse-test-results-${{ matrix.test }}.log
- exit 0
+
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/parse-test-results.sh
\
+ "$LOG_FILE"
+
+ status_code=$?
+ case $status_code in
+ 0) # All tests passed
+ echo "All tests passed successfully"
+ ;;
+ 1) # Tests failed but parsed successfully
+ echo "Test failures detected but properly parsed"
+ # Don't exit here - let the workflow continue
+ ;;
+ 2) # Parse error or missing file
+ echo "::warning::Could not parse test results properly"
+ {
+ echo "status=parse_error"
+ echo "total_tests=0"
+ echo "failed_tests=0"
+ echo "passed_tests=0"
+ echo "ignored_tests=0"
+ } >> "$GITHUB_OUTPUT"
+ exit 0
+ ;;
+ *) # Unexpected error
+ echo "::warning::Unexpected error during test results
parsing"
+ {
+ echo "status=unknown_error"
+ echo "total_tests=0"
+ echo "failed_tests=0"
+ echo "passed_tests=0"
+ echo "ignored_tests=0"
+ } >> "$GITHUB_OUTPUT"
+ exit 0
+ ;;
+ esac
+ } 2>&1 | tee -a build-logs/details/parse-test-results-${{
matrix.test }}.log
+
+ - name: Check and Display Regression Diffs
+ if: always()
+ run: |
+ # Search for regression.diffs recursively
+ found_file=$(find . -type f -name "regression.diffs" | head -n 1)
+ if [[ -n "$found_file" ]]; then
+ echo "Found regression.diffs at: $found_file"
+ cat "$found_file"
+ else
+ echo "No regression.diffs file found in the hierarchy."
fi
- chmod +x
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/parse-test-results.sh
-
- MAKE_NAME='${{ matrix.test }}' \
-
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/parse-test-results.sh
\
- build-logs/details/make-${{ matrix.test }}.log
- status_code=$?
-
- echo "status_code=$status_code"
-
- case $status_code in
- 0) # All tests passed
- echo "All tests passed successfully"
- ;;
- 1) # Tests failed but parsed successfully
- echo "Test failures detected but properly parsed"
- # Don't exit here - let the workflow continue
- ;;
- 2) # Parse error or missing file
- echo "::error::Could not find or access test results file"
- exit 1
- ;;
- *) # Unexpected error
- echo "::error::Unexpected error during test results parsing"
- exit 1
- ;;
- esac
-
- # Debug: verify outputs were set
- echo "Debug: Checking GITHUB_OUTPUT contents:"
- cat $GITHUB_OUTPUT || true
-
- } 2>&1 | tee -a build-logs/details/parse-test-results-${{ matrix.test
}}.log
-
- - name: "Generate Test Job Summary End: ${{ matrix.test }}"
- if: always()
- shell: bash {0}
- run: |
- {
- if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
- echo "## Test Results - SKIPPED"
+ - name: "Generate Test Job Summary End: ${{ matrix.test }}"
+ if: always()
+ shell: bash {0}
+ run: |
+ {
+ if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
+ echo "## Test Results - SKIPPED"
+ echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
+ exit 0
+ fi
+
+ echo "## Test Results"
echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
- exit 0
- fi
- echo "## Test Results"
- echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
-
- # Add test execution results
- echo "### Test Execution Summary"
- echo "| Metric | Count |"
- echo "|--------|-------|"
- echo "| Total Tests | ${{ steps.test-results.outputs.total_tests }}
|"
- echo "| Passed Tests | ${{ steps.test-results.outputs.passed_tests
}} |"
- echo "| Failed Tests | ${{ steps.test-results.outputs.failed_tests
}} |"
-
- # Test status with emoji based on results
- echo "### Test Status"
- if [[ "${{ steps.test-results.outputs.status }}" == "passed" ]]; then
- echo "✅ All ${{ steps.test-results.outputs.total_tests }} tests
passed successfully"
- else
- echo "⚠️ ${{ steps.test-results.outputs.failed_tests }} of ${{
steps.test-results.outputs.total_tests }} tests failed"
+ # Check if job was cancelled
+ if [[ "${{ job.status }}" == "cancelled" ]]; then
+ echo "### Test Status"
+ echo "🚫 Test execution was cancelled"
+ echo ""
+ echo "### Execution Summary"
+ echo "Test run was interrupted and did not complete. No test
results are available."
+ exit 0
+ fi
- # Add failed tests as a bullet list if there are failures
- if [[ -n "${{ steps.test-results.outputs.failed_test_names }}" ]];
then
+ # Add test execution results
+ echo "### Test Execution Summary"
+ echo "| Metric | Count |"
+ echo "|--------|-------|"
+ echo "| Total Tests | ${{ steps.test-results.outputs.total_tests
}} |"
+ echo "| Passed Tests | ${{ steps.test-results.outputs.passed_tests
}} |"
+ echo "| Failed Tests | ${{ steps.test-results.outputs.failed_tests
}} |"
+ echo "| Ignored Tests | ${{
steps.test-results.outputs.ignored_tests }} |"
+
+ # Test status with emoji based on results
+ echo "### Test Status"
+ if [[ "${{ steps.test-results.outputs.status }}" == "passed" ]];
then
+ echo "✅ All ${{ steps.test-results.outputs.total_tests }} tests
passed successfully"
+ else
+ echo "⚠️ ${{ steps.test-results.outputs.failed_tests }} of ${{
steps.test-results.outputs.total_tests }} tests failed"
+
+ # Add failed tests as a bullet list if there are failures
+ if [[ -n "${{ steps.test-results.outputs.failed_test_names }}"
]]; then
+ echo ""
+ echo "### Failed Tests"
+ echo "The following tests failed:"
+ echo "${{ steps.test-results.outputs.failed_test_names }}" |
tr ',' '\n' | while read -r test; do
+ if [[ -n "$test" ]]; then
+ echo "* \`${test}\`"
+ fi
+ done
+ fi
+ fi
+
+ # Add ignored tests as a separate section
+ if [[ -n "${{ steps.test-results.outputs.ignored_test_names }}"
]]; then
echo ""
- echo "### Failed Tests"
- echo "The following tests failed:"
- echo "${{ steps.test-results.outputs.failed_test_names }}" | tr
',' '\n' | while read -r test; do
+ echo "### Ignored Tests"
+ echo "The following tests were ignored (results not critical):"
+ echo "${{ steps.test-results.outputs.ignored_test_names }}" | tr
',' '\n' | while read -r test; do
if [[ -n "$test" ]]; then
echo "* \`${test}\`"
fi
done
fi
- fi
- } >> $GITHUB_STEP_SUMMARY || true
-
- - name: Upload test logs
- if: always()
- uses: actions/upload-artifact@v4
- with:
- name: test-logs-${{ matrix.test }}-${{
needs.build.outputs.build_timestamp }}
- path: |
- build-logs/
- **/regression.diffs
- **/regression.out
- retention-days: ${{ env.LOG_RETENTION_DAYS }}
-
- - name: Debug Upload Condition
- if: always()
- run: |
- echo "Upload condition check:"
- echo "status: '${{ steps.test-results.outputs.status }}'"
- echo "condition result: ${{ steps.test-results.outputs.status ==
'failed' }}"
-
- - name: Upload test regression logs
- if: always() && steps.test-results.outputs.status == 'failed'
- uses: actions/upload-artifact@v4
- with:
- name: regression-logs-${{ matrix.test }}-${{
needs.build.outputs.build_timestamp }}
- path: |
- src/test/regress/regression.out
- src/test/regress/regression.diffs
- src/test/regress/results/
- src/test/regress/expected/
- src/test/isolation2/regression.out
- src/test/isolation2/regression.diffs
- src/test/isolation2/results/
- src/test/isolation2/expected/
- gpAux/gpdemo/datadirs/standby/log/
- gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/
- gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/log/
- gpAux/gpdemo/datadirs/dbfast2/demoDataDir1/log/
- gpAux/gpdemo/datadirs/dbfast3/demoDataDir2/log/
- gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/log/
- gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1/log/
- gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2/log/
- retention-days: ${{ env.LOG_RETENTION_DAYS }}
+
+ } >> "$GITHUB_STEP_SUMMARY" || true
+
+ - name: Upload test logs
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: test-logs-${{ matrix.test }}-${{
needs.build.outputs.build_timestamp }}
+ path: |
+ build-logs/
+ retention-days: ${{ env.LOG_RETENTION_DAYS }}
+
+ - name: Upload test regression logs
+ if: >-
+ always() && (
+ steps.test-results.outputs.status == 'failed' ||
+ job.status == 'cancelled' ||
+ steps.test-results.outputs.status == 'cancelled'
+ )
+ uses: actions/upload-artifact@v4
+ with:
+ name: regression-logs-${{ matrix.test }}-${{
needs.build.outputs.build_timestamp }}
+ path: |
+ src/test/**/regression.out
+ src/test/**/regression.diffs
+ src/test/**/results/
+ gpAux/gpdemo/datadirs/standby/log/
+ gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/
+ gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/log/
+ gpAux/gpdemo/datadirs/dbfast2/demoDataDir1/log/
+ gpAux/gpdemo/datadirs/dbfast3/demoDataDir2/log/
+ gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/log/
+ gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1/log/
+ gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2/log/
+ retention-days: ${{ env.LOG_RETENTION_DAYS }}
report:
name: Generate Apache Cloudberry Build Report
@@ -1072,35 +1325,35 @@ jobs:
- name: Generate Final Report
run: |
{
- echo "# Apache Cloudberry Build Pipeline Report" >>
$GITHUB_STEP_SUMMARY
+ echo "# Apache Cloudberry Build Pipeline Report"
if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]];
then
- echo "## CI Skip Status" >> $GITHUB_STEP_SUMMARY
- echo "✅ CI checks skipped via skip flag" >> $GITHUB_STEP_SUMMARY
- echo "- Completion Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')" >>
$GITHUB_STEP_SUMMARY
+ echo "## CI Skip Status"
+ echo "✅ CI checks skipped via skip flag"
+ echo "- Completion Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
else
- echo "## Job Status" >> $GITHUB_STEP_SUMMARY
- echo "- Build Job: ${{ needs.build.result }}" >>
$GITHUB_STEP_SUMMARY
- echo "- Test Job: ${{ needs.test.result }}" >>
$GITHUB_STEP_SUMMARY
- echo "- Completion Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')" >>
$GITHUB_STEP_SUMMARY
+ echo "## Job Status"
+ echo "- Build Job: ${{ needs.build.result }}"
+ echo "- Test Job: ${{ needs.test.result }}"
+ echo "- Completion Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
if [[ "${{ needs.build.result }}" == "success" && "${{
needs.test.result }}" == "success" ]]; then
- echo "✅ Pipeline completed successfully" >>
$GITHUB_STEP_SUMMARY
+ echo "✅ Pipeline completed successfully"
else
- echo "⚠️ Pipeline completed with failures" >>
$GITHUB_STEP_SUMMARY
+ echo "⚠️ Pipeline completed with failures"
if [[ "${{ needs.build.result }}" != "success" ]]; then
- echo "### Build Job Failure" >> $GITHUB_STEP_SUMMARY
- echo "Check build logs for details" >> $GITHUB_STEP_SUMMARY
+ echo "### Build Job Failure"
+ echo "Check build logs for details"
fi
if [[ "${{ needs.test.result }}" != "success" ]]; then
- echo "### Test Job Failure" >> $GITHUB_STEP_SUMMARY
- echo "Check test logs and regression files for details" >>
$GITHUB_STEP_SUMMARY
+ echo "### Test Job Failure"
+ echo "Check test logs and regression files for details"
fi
fi
fi
- }
+ } >> "$GITHUB_STEP_SUMMARY"
- name: Notify on failure
if: |
diff --git
a/src/test/isolation2/expected/resgroup/resgroup_memory_limit_resgroup_v2.out
b/src/test/isolation2/expected/resgroup/resgroup_memory_limit_resgroup_v2.out
new file mode 100644
index 0000000000..1653733867
--- /dev/null
+++
b/src/test/isolation2/expected/resgroup/resgroup_memory_limit_resgroup_v2.out
@@ -0,0 +1,73 @@
+-- test memory limit
+-- start_ignore
+DROP TABLE IF EXISTS t_memory_limit;
+DROP
+DROP ROLE IF EXISTS role_memory_test;
+DROP
+DROP RESOURCE GROUP rg_memory_test;
+ERROR: resource group "rg_memory_test" does not exist
+-- end_ignore
+
+-- create a pl function to show the memory used by a process
+CREATE OR REPLACE FUNCTION func_memory_test (text) RETURNS text as /*in func*/
$$ /*in func*/ DECLARE /*in func*/ ln text; /*in func*/ tmp text[]; /*in func*/
match bool := false; /*in func*/ BEGIN /*in func*/ FOR ln IN execute
format('explain analyze %s', $1) LOOP /*in func*/ IF NOT match THEN /*in
func*/ tmp := regexp_match(ln, 'Memory used: (.*)'); /*in func*/ IF tmp IS NOT
null THEN /*in func*/ match := true; /*in func*/ END IF; /*in func*/ END IF;
/*in func*/ END LOOP; /*in fu [...]
+CREATE
+
+-- create a resource group with memory limit 100 Mb
+CREATE RESOURCE GROUP rg_memory_test WITH(memory_limit=100,
cpu_max_percent=20, concurrency=2);
+CREATE
+CREATE ROLE role_memory_test RESOURCE GROUP rg_memory_test;
+CREATE
+
+-- session1: explain memory used by query
+-- user requests less than statement_mem, set query's memory limit to
statement_mem
+1: SET ROLE TO role_memory_test;
+SET
+1: CREATE TABLE t_memory_limit(a int);
+CREATE
+1: BEGIN;
+BEGIN
+1: SHOW statement_mem;
+ statement_mem
+---------------
+ 125MB
+(1 row)
+1: SELECT func_memory_test('SELECT * FROM t_memory_limit');
+ func_memory_test
+------------------
+ 128000kB
+(1 row)
+
+-- session2: test alter resource group's memory limit
+2: ALTER RESOURCE GROUP rg_memory_test SET memory_limit 1000;
+ALTER
+
+-- memory used will grow up to 500 Mb
+1: SELECT func_memory_test('SELECT * FROM t_memory_limit');
+ func_memory_test
+------------------
+ 512000kB
+(1 row)
+1: END;
+END
+-- set gp_resgroup_memory_query_fixed_mem to 200MB
+1: SET gp_resgroup_memory_query_fixed_mem to 204800;
+SET
+1: SELECT func_memory_test('SELECT * FROM t_memory_limit');
+ func_memory_test
+------------------
+ 204800kB
+(1 row)
+1: RESET gp_resgroup_memory_query_fixed_mem;
+RESET
+
+1: RESET ROLE;
+RESET
+-- clean
+DROP FUNCTION func_memory_test(text);
+DROP
+DROP TABLE t_memory_limit;
+DROP
+DROP ROLE IF EXISTS role_memory_test;
+DROP
+DROP RESOURCE GROUP rg_memory_test;
+DROP
diff --git a/src/test/isolation2/isolation2_resgroup_v2_schedule
b/src/test/isolation2/isolation2_resgroup_v2_schedule
index 30425bba34..0190efcd3d 100644
--- a/src/test/isolation2/isolation2_resgroup_v2_schedule
+++ b/src/test/isolation2/isolation2_resgroup_v2_schedule
@@ -28,7 +28,7 @@ test: resgroup/resgroup_cpu_max_percent
test: resgroup/resgroup_cpuset
test: resgroup/resgroup_cpuset_empty_default
test: resgroup/resgroup_cancel_terminate_concurrency
-test: resgroup/resgroup_memory_limit
+test: resgroup/resgroup_memory_limit_resgroup_v2
test: resgroup/resgroup_move_query
test: resgroup/resgroup_io_limit
diff --git
a/src/test/isolation2/sql/resgroup/resgroup_memory_limit_resgroup_v2.sql
b/src/test/isolation2/sql/resgroup/resgroup_memory_limit_resgroup_v2.sql
new file mode 100644
index 0000000000..4db7eb0995
--- /dev/null
+++ b/src/test/isolation2/sql/resgroup/resgroup_memory_limit_resgroup_v2.sql
@@ -0,0 +1,57 @@
+-- test memory limit
+-- start_ignore
+DROP TABLE IF EXISTS t_memory_limit;
+DROP ROLE IF EXISTS role_memory_test;
+DROP RESOURCE GROUP rg_memory_test;
+-- end_ignore
+
+-- create a pl function to show the memory used by a process
+CREATE OR REPLACE FUNCTION func_memory_test (text) RETURNS text as /*in func*/
+$$ /*in func*/
+DECLARE /*in func*/
+ ln text; /*in func*/
+ tmp text[]; /*in func*/
+ match bool := false; /*in func*/
+BEGIN /*in func*/
+ FOR ln IN execute format('explain analyze %s', $1) LOOP /*in func*/
+ IF NOT match THEN /*in func*/
+ tmp := regexp_match(ln, 'Memory used: (.*)'); /*in
func*/
+ IF tmp IS NOT null THEN /*in func*/
+ match := true; /*in func*/
+ END IF; /*in func*/
+ END IF; /*in func*/
+ END LOOP; /*in func*/
+ RETURN tmp[1]; /*in func*/
+END; /*in func*/
+$$ /*in func*/
+LANGUAGE plpgsql;
+
+-- create a resource group with memory limit 100 Mb
+CREATE RESOURCE GROUP rg_memory_test WITH(memory_limit=100,
cpu_max_percent=20, concurrency=2);
+CREATE ROLE role_memory_test RESOURCE GROUP rg_memory_test;
+
+-- session1: explain memory used by query
+-- user requests less than statement_mem, set query's memory limit to
statement_mem
+1: SET ROLE TO role_memory_test;
+1: CREATE TABLE t_memory_limit(a int);
+1: BEGIN;
+1: SHOW statement_mem;
+1: SELECT func_memory_test('SELECT * FROM t_memory_limit');
+
+-- session2: test alter resource group's memory limit
+2: ALTER RESOURCE GROUP rg_memory_test SET memory_limit 1000;
+
+-- memory used will grow up to 500 Mb
+1: SELECT func_memory_test('SELECT * FROM t_memory_limit');
+1: END;
+-- set gp_resgroup_memory_query_fixed_mem to 200MB
+1: SET gp_resgroup_memory_query_fixed_mem to 204800;
+1: SELECT func_memory_test('SELECT * FROM t_memory_limit');
+1: RESET gp_resgroup_memory_query_fixed_mem;
+
+1: RESET ROLE;
+-- clean
+DROP FUNCTION func_memory_test(text);
+DROP TABLE t_memory_limit;
+DROP ROLE IF EXISTS role_memory_test;
+DROP RESOURCE GROUP rg_memory_test;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
