raulcd commented on PR #46922: URL: https://github.com/apache/arrow/pull/46922#issuecomment-3023842487
It is happening on buildkite, that's why I am trying to reproduce locally, see: https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/builds/1341/steps/canvas?sid=0197c5ba-f505-435b-a81c-1db7c21b4554 And the error: ``` INFO:buildkite.benchmark.run:{'type': 'BenchmarkGroupExecution', 'id': '06f7f943017147708e06ec94eeb77c3e', 'lang': 'C++', 'name': 'cpp-micro', 'options': '', 'flags': {'language': 'C++'}, 'benchmarkable_id': '2890f7d2eb77d13e7dddbe7cd870a16f44378059', 'run_id': '162b6a4f37174b7284c747a7af96b542', 'run_name': 'pull request: 46922', 'machine': 'amd64-c6a-4xlarge-linux', 'process_pid': 16243, 'command': 'conbench cpp-micro --run-id=$RUN_ID --run-name="$RUN_NAME" --run-reason="$RUN_REASON"', 'started_at': '2025-07-01 11:35:38.789532', 'finished_at': '2025-07-01 11:59:25.908120', 'total_run_time': '0:23:47.118588', 'failed': True, 'return_code': 1, 'stderr': '250701-11:35:39.290 INFO: Initializing adapter\n250701-11:35:39.327 INFO: Initializing adapter\n250701-11:35:39.327 INFO: Running benchmarks with command: `archery benchmark run --output /tmp/tmpm7_g9j3t --repetitions 6 --repetition-min-time 0.05 --suite-filter ^(?!arrow-acero-aggregate-benchmark)(?!arrow-filesystem-s3fs-benchmar k).*$ /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp`\n2025-07-01T11:35:45+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-arrow-reader-writer-benchmark\nRun on (16 X 2881.72 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 4.42, 7.01, 3.85\n2025-07-01T11:40:03+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-arrow-size-stats-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.05, 3.56, 3.16\n2025-07-01T11:40:11+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-page-index-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.04, 3.47, 3.14\n2025-07-01T11:40 :21+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-metadata-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.03, 3.39, 3.12\n2025-07-01T11:40:50+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-level-conversion-benchmark\nRun on (16 X 2870.82 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.02, 3.16, 3.05\n2025-07-01T11:40:51+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-encoding-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.02, 3.16, 3.05\n2025-07-01T11:43:51+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-co lumn-io-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.06, 2.19, 2.68\n2025-07-01T11:44:34+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-column-reader-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.03, 2.04, 2.61\n2025-07-01T11:45:07+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/parquet-bloom-filter-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.01, 1.92, 2.55\n2025-07-01T11:45:15+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-ipc-read-write-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x 8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.01, 1.89, 2.53\n2025-07-01T11:47:45+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-json-parser-benchmark\nRun on (16 X 3598.91 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.93, 1.81, 2.39\n2025-07-01T11:48:08+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-filesystem-localfs-benchmark\nRun on (16 X 2879.04 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.66, 1.76, 2.36\n2025-07-01T11:49:13+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-dataset-scanner-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.54, 1.71, 2.30\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\n2025-07-01T11:49:18+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-dataset-file-benchmark\nRun on (16 X 2879.38 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.73, 1.75, 2.31\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\n2025-07-01T11:49:22+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-acero-hash-join-benchmark\nRun on (16 X 2650 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 1.67, 1.73, 2.30\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\n2025-07-01T11:51:48+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release /arrow-acero-asof-join-benchmark\nRun on (16 X 2893.53 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 3.54, 2.18, 2.37\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\n2025-07-01T11:52:19+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-acero-expression-benchmark\nRun on (16 X 3599.07 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 2.54, 2.07, 2.32\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\n2025-07-01T11:54:13+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-acero-tpch-benchmark\nRun on (16 X 2886.59 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 13.25, 6.08, 3 .75\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\n2025-07-01T11:54:20+00:00\nRunning /tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-acero-project-benchmark\nRun on (16 X 2876.66 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB (x8)\n L1 Instruction 32 KiB (x8)\n L2 Unified 512 KiB (x8)\n L3 Unified 32768 KiB (x1)\nLoad Average: 13.47, 6.25, 3.81\n***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.\nTraceback (most recent call last):\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/archery", line 8, in <module>\n sys.exit(archery())\n ^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1442, in __call__\n return self.main(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1363, in main\n rv = sel f.invoke(ctx)\n ^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1830, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1830, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1226, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 794, in invoke\n return callback(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/. conda/envs/arrow-commit/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func\n return f(get_current_context(), *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/cli.py", line 469, in benchmark_run\n json_out = json.dumps(runner_base, cls=JsonEncoder)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/json/__init__.py", line 238, in dumps\n **kw).encode(obj)\n ^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/json/encoder.py", line 200, in encode\n chunks = self.iterencode(o, _one_shot=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/json/encoder.py", line 258, in iterencode\n return _iterencode(o, 0)\n ^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/codec.py", line 35, in default\n return BenchmarkRunnerCodec.encode(o)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/codec.py", line 79, in encode\n return {"suites": [BenchmarkSuiteCodec.encode(s) for s in br.suites]}\n ^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/runner.py", line 183, in suites\n suite = self.suite(suite_name, suite_bin)\ n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/runner.py", line 157, in suite\n results = suite_cmd.results(\n ^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/google.py", line 67, in results\n self.run(*argv, check=True)\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/utils/command.py", line 78, in run\n return subprocess.run(invocation, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/subprocess.py", line 571, in run\n ra ise CalledProcessError(retcode, process.args,\nsubprocess.CalledProcessError: Command \'[\'/tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp/release/arrow-acero-project-benchmark\', \'--benchmark_repetitions=6\', \'--benchmark_out=/tmp/tmptt7w0j9e\', \'--benchmark_out_format=json\', \'--benchmark_min_time=0.050000s\']\' died with <Signals.SIGSEGV: 11>.\nTraceback (most recent call last):\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/conbench", line 8, in <module>\n sys.exit(conbench())\n ^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1442, in __call__\n return self.main(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1363, in main\n rv = self.invoke(ctx)\n ^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-pack ages/click/core.py", line 1830, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1226, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 794, in invoke\n return callback(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/conbenchlegacy/cli.py", line 149, in _benchmark\n for result, output in benchmark().run(**kwargs):\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/benchmarks/benc hmarks/cpp_micro_benchmarks.py", line 136, in run\n self.adapter.run(command_params)\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/benchadapt/adapters/_adapter.py", line 98, in run\n subprocess.run(args=command, check=True)\n File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/subprocess.py", line 571, in run\n raise CalledProcessError(retcode, process.args,\nsubprocess.CalledProcessError: Command \'[\'archery\', \'benchmark\', \'run\', \'--output\', PosixPath(\'/tmp/tmpm7_g9j3t\'), \'--repetitions\', \'6\', \'--repetition-min-time\', \'0.05\', \'--suite-filter\', \'^(?!arrow-acero-aggregate-benchmark)(?!arrow-filesystem-s3fs-benchmark).*$\', \'/tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp\']\' returned non-zero exit status 1.\n', 'total_machine_virtual_memory': 32900587520} ======================= Benchmark Groups Results ========================== FAILED C++ cpp-micro 1 n-min-time', '0.05', '--suite-filter', '^(?!arrow-acero-aggregate-benchmark)(?!arrow-filesystem-s3fs-benchmark).*$', '/tmp/3e8eb6cb-f0cc-4e18-b89b-5f88fb1f2d2f/cpp']' returned non-zero exit status 1. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/buildkite/benchmark/run_benchmark_groups.py", line 13, in <module> run_benchmark_groups() File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/buildkite/benchmark/run_benchmark_groups.py", line 10, in run_benchmark_groups run.run_all_benchmark_groups() File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0d44e9c252c2eabea-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/buildkite/benchmark/run.py", line 673, in run_all_benchmark_groups raise Exception("Build has failed benchmarks.") Exception: Build has failed benchmarks. ``` Seems to be due to segfault: `died with <Signals.SIGSEGV: 11>` That's why I am confused, the commit that was successful on buildkite previously for this PR it's also segfaulting locally. I am happy to merge and investigate the segfault on a different issue if we are ok with that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
