GujaLomsadze commented on issue #47133:
URL: https://github.com/apache/arrow/issues/47133#issuecomment-5050674452
**TL;DR:** Not a memory leak. pyarrow infers `dict -> struct`, and a nested
dict here has
near-unique keys (**73,106 distinct / 74,270 rows**), so it builds a struct
with **~73k
fields**, which is O(rows x keys), about **5.4B cells / ~22GB**. The same
data as a `map`
does the full dataset in **0.03s / 231MB**.
**It's not pickle or pandas-specific.** (3-line repro
below). Question at the end: guardrail, docs, or working-as-intended?
```python
import pyarrow as pa
data = [{f"key_{i}": "x"} for i in range(10_000)] # 10k dicts, each a
unique key
pa.array(data) # -> struct<10000 fields>; bump to 100k and it eats all
your RAM
```
`struct_fields == N` exactly: every distinct key becomes a field.
<details>
<summary>Full script to reproduce this (click to expand)</summary>
```python
"""
Reproduce apache/arrow #47133: dict-with-unique-keys -> struct explosion.
"""
import time
import resource
import pyarrow as pa
import pandas as pd
def peak_rss_mb():
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
def make_dicts(n):
# each row = a dict with a single UNIQUE key -> n distinct keys total
return [{f"key_{i}": "x"} for i in range(n)]
print("pyarrow", pa.__version__, "| pandas", pd.__version__)
print("=" * 64)
# --- 1. Minimal repro: plain pyarrow, NO pandas ---
print("\n[1] no pandas: pa.array(list_of_dicts)")
data = make_dicts(10_000)
arr = pa.array(data)
print(" struct fields :", len(arr.type)) # 10000
print(" first 3 names :", [arr.type.field(i).name for i in range(3)])
print(" nbytes :", round(arr.nbytes / 1e6, 1), "MB")
# --- 2. Same thing via pandas from_pandas (how the report hit it) ---
print("\n[2] pandas: pa.Table.from_pandas(df)")
df = pd.DataFrame({"col": data})
tbl = pa.Table.from_pandas(df)
print(" struct fields :", len(tbl.schema.field("col").type))
print(" nbytes :", round(tbl.column("col").nbytes / 1e6, 1), "MB")
# --- 3a. Scaling WITHOUT pandas (pa.array) ---
print("\n[3a] scaling, no pandas (pa.array) -- doubling N ~= 4x nbytes")
print(f" {'N':>6} {'fields':>7} {'nbytes(MB)':>11} {'time(s)':>8}")
for n in (2000, 4000, 8000, 16000):
d = make_dicts(n)
t = time.time()
a = pa.array(d)
dt = time.time() - t
print(f" {n:>6} {len(a.type):>7} {a.nbytes / 1e6:>11.1f} {dt:>8.2f}")
# --- 3b. Scaling WITH pandas (from_pandas) ---
print("\n[3b] scaling, pandas (from_pandas) -- same quadratic curve")
print(f" {'N':>6} {'fields':>7} {'nbytes(MB)':>11} {'time(s)':>8}")
for n in (2000, 4000, 8000, 16000):
d = make_dicts(n)
df_n = pd.DataFrame({"col": d})
t = time.time()
t_n = pa.Table.from_pandas(df_n)
dt = time.time() - t
nb = t_n.column("col").nbytes / 1e6
print(f" {n:>6} {len(t_n.schema.field('col').type):>7} {nb:>11.1f}
{dt:>8.2f}")
# --- 4. The fix: it's really a map, not a struct (both paths) ---
print("\n[4] correct type: map<string, string> (cheap, no explosion)")
m_nopd = pa.array(data, type=pa.map_(pa.string(), pa.string()))
m_pd = pa.Table.from_pandas(
df, schema=pa.schema([("col", pa.map_(pa.string(), pa.string()))])
)
print(" no pandas nbytes :", round(m_nopd.nbytes / 1e6, 2), "MB")
print(" pandas nbytes :", round(m_pd.column("col").nbytes / 1e6, 2),
"MB")
print(" (vs struct's ~400 MB for the same 10k rows)")
print("\n" + "=" * 64)
print("peak RSS this process:", round(peak_rss_mb(), 1), "MB")
```
</details>
---
@Seminko got the pickle, thanks! Drive link still works lol. Reproduced and
root-caused -it's not a leak, it's schema
inference.
The url_parts dicts contain a nested queries dict whose keys are effectively
unique across
rows (73,106 distinct keys in 74,270 rows). PyArrow infers dict -> struct,
and a struct's
fields are the union of all keys, so queries becomes a struct with ~73k
fields. Materializing
that over 74k rows is O(rows x fields), about 5.4B cells, roughly 22GB
(almost all my RAM
haha). Batching to 10k "works" only because each batch sees far fewer
distinct keys.
IMO the right Arrow type for open-ended keys is a **map**, not a struct.
Converting the same
column as `map<string, list<string>>` does the full 74,270 rows in **0.03s /
231MB**.
**It's also not pickle- or pandas-specific. Pickle was just how the report
shipped the data. The
real trigger is any dict column with lots of distinct keys (JSON/API
payloads, scraped or
parsed URLs, NoSQL dumps, and so on). The 3-line repro above uses plain
pa.array, no pandas
and no pickle. from_pandas just forwards to this same inference, so a
guardrail at the
inference layer would cover all of these paths.**
So: not a leak!! inference just builds a pathological struct. A few ways
this could go, happy
to do whichever you prefer:
1. Guardrail: when an inferred struct exceeds some field count, raise a
clear error pointing
at pa.map_() instead of silently consuming entire RAM.
2. Docs: a short note on dict -> struct vs map, and when to use each.
3. Both.
4. ... something else, haha
Or if you'd rather treat this as working-as-intended, let me know the
preferred direction and I'll pick it up.
Way I look at it is that, at least it should drop some warning th
cc @AlenkaF @jorisvandenbossche - this is your area (`from_pandas`
inference).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
