stephen-a-stc opened a new issue, #54885:
URL: https://github.com/apache/arrow/issues/54885
### Describe the bug, including details regarding any error messages,
version, and platform.
## Summary
`pyarrow.Table.join()` crashes my Python session when 1 or more
`large_string` columns are used as keys, and the data in those columns nears
2gb.
Specifically, we can use the following inequality to define the bound of
where this bug occurs
> occurs when: `[num_rows]*[num_keys]*([key_length]+9) > [2 gigabytes]`
> In words: if the amount of key data plus the additional 9 bytes per key
exceeds 2gb, a fatal crash occurs.
I focused my investigation on 'left anti' joins, where the right table is
near boundary, in which case the left table's size seems irrelevant. Just
before submission, I confirmed that this occurs with all join types when tested
with a large right table.
### Expected Behaviour
The table join function should support the use of `large_string` columns of
any size as keys.
### Actual Behaviour
The join function causes a fatal crash when `large_string` key columns
exceed a certain size which is below, but near, 2gb.
## Tested With
PyArrow versions tested:
- `24.0.1`
- `25.0.0`
- `26.0.0`
Platform:
- Windows
- Linux
- Microsoft Fabric (Spark RT 2.0)
## Minimal Reproducer
```python
import faulthandler
import numpy as np
import pyarrow as pa
faulthandler.enable()
def make_join_test_data(
left_rows=1_000_000,
right_rows=50_000_000,
num_keys=3,
key_length=15,
):
def build_table(n_rows):
cols = {}
for k in range(num_keys):
cols[f"KEY{k+1}"] = pa.array(
[f"{i:0{key_length}d}" for i in range(n_rows)],
type=pa.large_string(),
)
cols["INTVAL"] = pa.array(np.arange(n_rows, dtype=np.int64))
cols["DOUBLEVAL"] = pa.array(np.arange(n_rows, dtype=np.float64))
return pa.table(cols)
left = build_table(left_rows)
right = build_table(right_rows)
return left, right
left, right = make_join_test_data(
right_rows=534_865 + 1, # remove the `+ 1` and it will work
left_rows=10_000,
num_keys=5,
key_length=794,
)
join_keys = [col for col in right.column_names if col.startswith("KEY")]
print(f"joining with {join_keys}")
left.join(right, keys=join_keys, join_type="left anti") # BUG: note that it
fails with all join types
print("it worked")
```
### Output (pyarrow 25.0.0 cp314-cp314-linux_x86_64 )
```plaintext
joining with ['KEY1', 'KEY2', 'KEY3', 'KEY4', 'KEY5']
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
Fatal Python error: Aborted
Thread 0x00007fdc27af7740 [python] (most recent call first):
File "/opt/conda/lib/python3.14/site-packages/pyarrow/acero.py", line 253
in _perform_join
File "/home/jovyan/try_repro.py", line 39 in <module>
Current thread's C stack trace (most recent call first):
Binary file "python", at _Py_DumpStack+0x4a [0x56532cc3e4ba]
Binary file "python", at +0x18f928 [0x56532cc3e928]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x45330 [0x7fdc27b41330]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at pthread_kill+0x11c
[0x7fdc27b9ac0c]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at gsignal+0x1e
[0x7fdc27b4127e]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at abort+0xdf
[0x7fdc27b248ff]
Binary file
"/opt/conda/lib/python3.14/site-packages/numpy/_core/../../../../libstdc++.so.6",
at +0xb2cd2 [0x7fdc25b17cd2]
Binary file
"/opt/conda/lib/python3.14/site-packages/numpy/_core/../../../../libstdc++.so.6",
at +0xc5a1a [0x7fdc25b2aa1a]
Binary file
"/opt/conda/lib/python3.14/site-packages/numpy/_core/../../../../libstdc++.so.6",
at _ZSt10unexpectedv+0x0 [0x7fdc25b1784d]
Binary file
"/opt/conda/lib/python3.14/site-packages/numpy/_core/../../../../libstdc++.so.6",
at +0xc5cb7 [0x7fdc25b2acb7]
Binary file
"/opt/conda/lib/python3.14/site-packages/numpy/_core/../../../../libstdc++.so.6",
at _ZSt20__throw_length_errorPKc+0x38 [0x7fdc25b1abe1]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_compute.so.2500",
at +0x11b30d [0x7fdc1800430d]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_compute.so.2500",
at
_ZN5arrow7compute8internal10RowEncoder15EncodeAndAppendERKNS0_8ExecSpanE+0x13a
[0x7fdc188f627a]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_acero.so.2500",
at
_ZNK5arrow5acero22HashJoinDictBuildMulti11EncodeBatchEmRKNS0_20SchemaProjectionMapsINS0_18HashJoinProjectionEEERKNS_7compute9ExecBatchEPNS7_8internal10RowEncoderEPNS7_11ExecContextE+0x1a6
[0x7fdc00a10126]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_acero.so.2500",
at +0x9e9cf [0x7fdc00a0a9cf]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_acero.so.2500",
at _ZN5arrow5acero17TaskSchedulerImpl11ExecuteTaskEmilPb+0x69 [0x7fdc00a5abc9]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_acero.so.2500",
at +0xfa151 [0x7fdc00a66151]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_acero.so.2500",
at +0xc6d15 [0x7fdc00a32d15]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow_acero.so.2500",
at +0xbf557 [0x7fdc00a2b557]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow.so.2500", at
+0xa57be2 [0x7fdc195fbbe2]
Binary file
"/opt/conda/lib/python3.14/site-packages/pyarrow/../../../libarrow.so.2500", at
+0xa5499c [0x7fdc195f899c]
Binary file
"/opt/conda/lib/python3.14/site-packages/numpy/_core/../../../../libstdc++.so.6",
at +0xf5b86 [0x7fdc25b5ab86]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x9cb84 [0x7fdc27b98b84]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x129ecc
[0x7fdc27c25ecc]
Extension modules: numpy._core._multiarray_umath,
numpy.linalg._umath_linalg, pyarrow.lib, numpy.random._common,
numpy.random.bit_generator, numpy.random._bounded_integers,
numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937,
numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cyutility,
pandas._libs._cyutility, pandas._libs.tslibs.ccalendar,
pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes,
pandas._libs.tslibs.base, pandas._libs.tslibs.nattype,
pandas._libs.tslibs.timezones, pandas._libs.properties,
pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas,
pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps,
pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime,
pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion,
pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized,
pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable,
pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._
compute, pandas._libs.ops, numexpr.interpreter, bottleneck.move,
bottleneck.nonreduce, bottleneck.nonreduce_axis, bottleneck.reduce,
pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib,
pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing,
pandas._libs.index, pandas._libs.writers, pandas._libs.join,
pandas._libs.window.aggregations, pandas._libs.window.indexers,
pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json,
pandas._libs.parsers, pandas._libs.testing, pyarrow._acero, pyarrow._fs,
pyarrow._csv, pyarrow._json, pyarrow._substrait, pyarrow._dataset,
pyarrow._dataset_orc, pyarrow._parquet, pyarrow._azurefs, pyarrow._hdfs,
pyarrow._gcsfs, pyarrow._s3fs, pyarrow._parquet_encryption,
pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet (total: 75)
Aborted
```
## Related Issues
- https://github.com/apache/arrow/issues/33151
- https://github.com/apache/arrow/issues/36879
## Investigation
I suspect there is an `int32_t` where there may need to be something larger.
The stack trace leads to the C++ method
`arrow.compute.internal.RowEncoder.EncodeAndAppend()` (found in
[row_encoder_internal.cc](https://github.com/apache/arrow/blob/1776578067602ed059db9bd8c3e4fc126c4597f3/cpp/src/arrow/compute/row/row_encoder_internal.cc#L358)).
This method makes use of a private member, `offsets_`, which is defined in
[row_encoder_internal.h](https://github.com/apache/arrow/blob/1776578067602ed059db9bd8c3e4fc126c4597f3/cpp/src/arrow/compute/row/row_encoder_internal.h#L397)
and contains the following note
```cpp
// offsets_ vector stores the starting position (offset) of each encoded row
// within the bytes_ vector. This allows for quick access to individual
rows.
//
// The size would be num_rows + 1 if not empty, the last element is the
total
// length of the bytes_ vector.
std::vector<int32_t> offsets_;
```
This has me wondering, if the bytes vector exceeds a certain size, will the
offset not be able to address it?
### Additional Reproductions
I ran a number of different tests with varying numbers of keys, key length,
rows of data, and different left table sizes. The following code documents
that and makes it easy to continue testing.
```python
"""pyarrow.Table.join fails under certain conditions.
pyarrow.Table.join fails when
- type is 'left anti'
- keys (1 or more) are `large_string`
- the volume of string key data + metadata exceeds a threshold of 2gb
The failure is not raised as an arrow exception but leads to 'Fatal Python
error: Aborted'.
The C Stack Trace leads to C++ method
`arrow.compute.internal.RowEncoder.EncodeAndAppend()`.
"""
import faulthandler
from pprint import pprint
import numpy as np
import pyarrow as pa
faulthandler.enable()
def print_table_summary(name, t, keys=None):
msg = f"\n{name}\n"
msg += f"rows : {t.num_rows:,}\n"
msg += f"columns : {t.num_columns}\n"
msg += f"nbytes : {t.nbytes} ({t.nbytes / 1024**3:.2f} GiB)\n"
key_bytes=sum([col.nbytes for col in t.select(keys).columns])
msg += f"key nbytes: {key_bytes} ({key_bytes / 1024**3:.2f} GiB)\n"
msg += f"chunks : {sum(len(c.chunks) for c in t.columns):,}\n"
for col_name in t.column_names:
col = t[col_name]
msg += f" {col_name:10}"
msg += f"{str(col.type):14}"
msg += f"chunks={len(col.chunks):<6,}"
msg += f" bytes={col.nbytes:-16}\n"
print(msg)
def make_join_test_data(
left_rows=1_000_000,
right_rows=50_000_000,
num_keys=3,
key_length=15,
final_key_length_adjustment=0,
):
"""Build tables as specified, each also includes 1 int and 1 double
column."""
print("building test data with specs:")
pprint(locals())
print()
# determine expected outcome
_2gb = 2*1024*1024*1024
ptr_bytes=8
null_bytes=1
est_bytes = right_rows*num_keys*(key_length + ptr_bytes + null_bytes)
if final_key_length_adjustment != 0:
est_bytes += final_key_length_adjustment*num_keys
est_buffer_free = _2gb - est_bytes
if est_buffer_free <0:
print("Expected outcome: FAILURE")
print (" Estimated join bytes exceed 2gb by ", -1*est_buffer_free,
" bytes")
else:
print("Expected outcome: SUCCESS")
print(" Join should pass with ", est_buffer_free, " bytes of extra
space")
def build_table(n_rows):
cols = {}
temp_col=[f"{i:0{key_length}d}" for i in range(n_rows)]
temp_col[-1] =
f"{(n_rows-1):0{key_length+final_key_length_adjustment}d}"
for k in range(num_keys):
cols[f"KEY{k+1}"] = pa.array(
temp_col,
type=pa.large_string(),
)
cols["INTVAL"] = pa.array(np.arange(n_rows, dtype=np.int64))
cols["DOUBLEVAL"] = pa.array(np.arange(n_rows, dtype=np.float64))
return pa.table(cols)
left = build_table(left_rows)
right = build_table(right_rows)
return left, right
# list of different shape tables at the point of failure
runs = [
{#0
"triggers_bug": False,
"args": {
"right_rows": 37_025_580,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 20,
},
},
{#1
"triggers_bug": True,
"args": {
"right_rows": 37_025_580 + 1,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 20,
},
},
{#2
"triggers_bug": False,
"args": {
"right_rows": 534_865,
"left_rows": 10_000,
"num_keys": 5,
"key_length": 794,
},
},
{#3
"triggers_bug": True,
"args": {
"right_rows": 534_865 + 1,
"left_rows": 10_000,
"num_keys": 5,
"key_length": 794,
},
},
{#4
"triggers_bug": False,
"args": {
"right_rows": 1_337_162,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 794,
},
},
{#5
"triggers_bug": True,
"args": {
"right_rows": 1_337_162 + 1,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 794,
},
},
{#6
"triggers_bug": False,
"args": {
"right_rows": 2_674_325,
"left_rows": 10_000,
"num_keys": 1,
"key_length": 794,
},
},
{#7
"triggers_bug": True,
"args": {
"right_rows": 2_674_325 + 1,
"left_rows": 10_000,
"num_keys": 1,
"key_length": 794,
},
},
{#8
"triggers_bug": False,
"args": {
"right_rows": 37_025_580,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 20,
"final_key_length_adjustment": 3,
},
},
{#9
"triggers_bug": True,
"args": {
"right_rows": 37_025_580,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 20,
"final_key_length_adjustment": 4,
},
},
{#10
"triggers_bug": False,
"args": {
"right_rows": 534_465,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 2000,
},
},
{#11
"triggers_bug": True,
"args": {
"right_rows": 534_465 + 1,
"left_rows": 10_000,
"num_keys": 2,
"key_length": 2000,
},
},
]
#left, right = make_join_test_data(
# left_rows=10_000,
# right_rows=37_025_580+1,
# num_keys=2,
# key_length=20,
# #final_key_length_adjustment=3
#)
left, right = make_join_test_data(
**runs[4]['args']
)
join_keys = [col for col in right.column_names if col.startswith("KEY")]
print_table_summary("left", left, join_keys)
print_table_summary("right", right, join_keys)
print(f"joining with {join_keys}")
# .join doesn't like too much `large_string` data
left.join(
right,
keys=join_keys,
join_type="left anti",
)
print("it worked")
```
#### Sample Output (`runs[3]`)
```plaintext
building test data with specs:
{'final_key_length_adjustment': 0,
'key_length': 794,
'left_rows': 10000,
'num_keys': 5,
'right_rows': 534866}
Expected outcome: FAILURE
Estimated join bytes exceed 2gb by 3342 bytes
left
rows : 10,000
columns : 7
nbytes : 40260000 (0.04 GiB)
key nbytes: 40100000 (0.04 GiB)
chunks : 7
KEY1 large_string chunks=1 bytes= 8020000
KEY2 large_string chunks=1 bytes= 8020000
KEY3 large_string chunks=1 bytes= 8020000
KEY4 large_string chunks=1 bytes= 8020000
KEY5 large_string chunks=1 bytes= 8020000
INTVAL int64 chunks=1 bytes= 80000
DOUBLEVAL double chunks=1 bytes= 80000
right
rows : 534,866
columns : 7
nbytes : 2153370516 (2.01 GiB)
key nbytes: 2144812660 (2.00 GiB)
chunks : 7
KEY1 large_string chunks=1 bytes= 428962532
KEY2 large_string chunks=1 bytes= 428962532
KEY3 large_string chunks=1 bytes= 428962532
KEY4 large_string chunks=1 bytes= 428962532
KEY5 large_string chunks=1 bytes= 428962532
INTVAL int64 chunks=1 bytes= 4278928
DOUBLEVAL double chunks=1 bytes= 4278928
joining with ['KEY1', 'KEY2', 'KEY3', 'KEY4', 'KEY5']
Fatal Python error: Aborted
Thread 0x00000c14 (most recent call first):
.
.
.
```
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
