robertHowlett opened a new issue, #50614:
URL: https://github.com/apache/arrow/issues/50614
### Describe the bug, including details regarding any error messages,
version, and platform.
## Description
We are seeing an intermittent bug with pyarrow 25.0.0 (but not pyarrow
24.0.0). Our code looks like
```
import io
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
import boto3
import pandas as pd
# Define AWS resources & lambda client to trigger functions
s3 = boto3.client(service_name="s3",
region_name=os.environ.get("AWS_REGION"))
def query(bucket: str, matching_objects: list[str]) -> dict[str,
pd.DataFrame]:
"""
Query a time-series parquet file using dynamodb
Args:
bucket: S3 bucket name
matching_objects: list of object keys e.g.
["1763466120/1234.parquet.snappy", "1763466120/1235.parquet.snappy"]
Returns:
Dictionary of DataFrames with object paths as the dictionary key
"""
with ThreadPoolExecutor(max_workers=None) as executor: # will default
to ncpus * 5 if max_workers = None
futures = {executor.submit(low_level_query, bucket=bucket,
s3_key=s3_key): s3_key for s3_key in matching_objects}
raw = {}
for future in as_completed(futures):
s3_key = futures[future]
try:
raw[s3_key] = future.result()
except Exception:
print(f"Error when querying {s3_key} from {bucket}")
continue
print(raw) # shows problems
return raw
def low_level_query(bucket: str, s3_key: str) -> pd.DataFrame:
"""
Query the given parquet file in an S3 bucket
Args:
bucket: bucket name
s3_key: S3 key (e.g. "1763466120/1234.parquet.snappy")
Returns:
DataFrame filtered by start and end timestamps
"""
# Load parquet from S3
data = pd.read_parquet(path=io.BytesIO(s3.get_object(Bucket=bucket,
Key=s3_key)["Body"].read()))
# Remove duplicates, keeping the last one
return data[~data.index.duplicated(keep="last")]
```
Maybe 60% of the time this works as expected, returning a list of
dataframes, but sometimes rows disappear. e.g. I get a df like
```
EXPORT_TARIFF IMPORT_TARIFF
TIMESTAMP
1784797200 12 26.38
1784799000 12 26.38
1784800800 12 26.38
1784809800 12 26.38
1784811600 12 26.38
1784813400 12 26.38
1784815200 12 26.38
1784795400 12 26.38 # Out of order
1784853000 12 26.38
1784854800 12 26.38
1784856600 12 26.38
1784858400 12 26.38
1784860200 12 26.38
1784862000 12 26.38
```
but I expect a df like
```
EXPORT_TARIFF IMPORT_TARIFF
TIMESTAMP
1784795400 12 26.38 # First row is moved
1784797200 12 26.38
1784799000 12 26.38
1784800800 12 26.38
1784802600 12 26.38
1784804400 12 26.38
1784806200 12 26.38
1784808000 12 26.38
1784809800 12 26.38
1784811600 12 26.38
1784813400 12 26.38
1784815200 12 26.38
1784817000 12 26.38 # First missing row
1784818800 12 26.38
1784820600 12 26.38
1784822400 12 26.38
1784824200 12 26.38
1784826000 12 26.38
1784827800 12 26.38
1784829600 12 26.38
1784831400 12 26.38
1784833200 12 26.38
1784835000 12 26.38
1784836800 12 26.38
1784838600 12 26.38
1784840400 12 26.38
1784842200 12 26.38
1784844000 12 26.38
1784845800 12 26.38
1784847600 12 26.38
1784849400 12 26.38
1784851200 12 26.38 # Last missing row
1784853000 12 26.38
1784854800 12 26.38
1784856600 12 26.38
1784858400 12 26.38
1784860200 12 26.38
1784862000 12 26.38
```
This I got by manually downloading the parquet from S3, then using
`pd.read_parquet("97.parquet.snappy")` in a console with my venv loaded. I get
the second result regardless of whether I use pyarrow 24 or 25.
## Tech stack
* AWS lambda (ARM64)
* Data is stored on AWS S3
* Python 3.14
* pandas 3.0.3
* pyarrow 25.0.0
## Other notes
* The failure pattern isn't uncorrelated, it starts failing for a while,
then stops failing.
<img width="1383" height="448" alt="Image"
src="https://github.com/user-attachments/assets/a6624d6f-9b53-4f60-9b38-df17c47c94ec";
/>
* I am sorry for the lack of detail/reproducability
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
