chaitanya588 opened a new issue, #10316:
URL: https://github.com/apache/arrow-rs/issues/10316
## Parquet files with page indexes written by parquet-rs are unreadable by
parquet-mr (Spark/Trino/Hive return 0 rows)
### Describe the bug
Parquet files written by `parquet-rs` (version 58.3.0+) with page indexes
(column index + offset index) return **0 rows** when read by Java-based readers
(`parquet-mr`), including Apache Spark, Trino (Athena), Redshift Spectrum, and
Hive. The same files are read correctly by `parquet-cpp` / pyarrow.
Reading the same data back with pyarrow and writing a new file (which does
NOT include page indexes) produces a file that all readers can consume
correctly.
### Root Cause
`parquet-rs` writes page indexes (column index + offset index) between the
row group data and the footer. When `parquet-mr` encounters these page indexes,
it returns 0 rows — likely misinterpreting the page index metadata during page
pruning.
Evidence:
- Original file (parquet-rs): **982 bytes** of page index data between row
group end and footer → **0 rows** in Spark/Trino
- Rewritten file (pyarrow): **0 bytes** gap (no page indexes) → **342 rows**
in Spark/Trino
- Both files contain identical data and use ZSTD compression
### To Reproduce
**1. Write a parquet file with parquet-rs (Rust):**
```rust
use arrow::array::{Int64Array, StringArray, TimestampMillisecondArray};
use arrow::datatypes::{DataType, Field, Schema, TimeUnit};
use arrow::record_batch::RecordBatch;
use parquet::arrow::ArrowWriter;
use parquet::basic::{Compression, ZstdLevel};
use parquet::file::properties::WriterProperties;
use std::fs::File;
use std::sync::Arc;
fn main() {
let schema = Arc::new(Schema::new(vec![
Field::new("node_ip", DataType::Utf8, true),
Field::new("level", DataType::Utf8, true),
Field::new("message", DataType::Utf8, true),
Field::new("timestamp", DataType::Timestamp(TimeUnit::Millisecond,
None), true),
Field::new("seq_no", DataType::Int64, false),
]));
let node_ips = StringArray::from(vec!["10.0.0.1", "10.0.0.2",
"10.0.0.1", "10.0.0.3", "10.0.0.2"]);
let levels = StringArray::from(vec!["ERROR", "WARN", "INFO", "ERROR",
"WARN"]);
let messages = StringArray::from(vec![
"Connection timeout", "High memory", "Request completed",
"Disk full", "Slow query detected",
]);
let timestamps = TimestampMillisecondArray::from(vec![
1718000000000i64, 1718000060000, 1718000120000, 1718000180000,
1718000240000,
]);
let seq_nos = Int64Array::from(vec![1i64, 2, 3, 4, 5]);
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(node_ips),
Arc::new(levels),
Arc::new(messages),
Arc::new(timestamps),
Arc::new(seq_nos),
],
).unwrap();
let file = File::create("test_parquet_rs.parquet").unwrap();
let props = WriterProperties::builder()
.set_compression(Compression::ZSTD(ZstdLevel::try_new(3).unwrap()))
.set_statistics_enabled(parquet::file::properties::EnabledStatistics::Page)
.build();
let mut writer = ArrowWriter::try_new(file, schema,
Some(props)).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
println!("Written: test_parquet_rs.parquet");
}
```
**2. Verify the file has page indexes:**
```python
import pyarrow.parquet as pq
import struct
pf = pq.ParquetFile("test_parquet_rs.parquet")
print(f"Rows: {pf.metadata.num_rows}") # Shows correct row count
# Check for page index gap
with open("test_parquet_rs.parquet", "rb") as f:
f.seek(-8, 2)
footer_len = struct.unpack('<I', f.read(4))[0]
file_size = f.seek(0, 2)
footer_start = file_size - 8 - footer_len
rg = pf.metadata.row_group(0)
last_col = rg.column(rg.num_columns - 1)
data_end = last_col.data_page_offset + last_col.total_compressed_size
gap = footer_start - data_end
print(f"Page index gap: {gap} bytes") # > 0 means page indexes present
```
**3. Try reading with Spark (returns 0 rows):**
```python
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.parquet("test_parquet_rs.parquet")
print(f"Spark row count: {df.count()}") # Prints 0!
```
**4. Verify pyarrow reads it correctly:**
```python
import pyarrow.parquet as pq
table = pq.read_table("test_parquet_rs.parquet")
print(f"pyarrow row count: {table.num_rows}") # Prints 5 (correct)
```
**5. Workaround — rewrite without page indexes:**
```python
import pyarrow.parquet as pq
table = pq.read_table("test_parquet_rs.parquet")
pq.write_table(table, "test_rewritten.parquet", compression="zstd")
# test_rewritten.parquet is now readable by Spark/Trino/Hive
```
### Affected Readers (all Java parquet-mr based)
| Reader | Result |
|--------|--------|
| Apache Spark 3.x/4.x | ❌ 0 rows |
| Trino (Athena v3) | ❌ 0 rows |
| AWS Redshift Spectrum | ❌ 0 rows |
| AWS Glue DataBrew | ❌ Footer read error |
| Apache Hive | ❌ 0 rows (inferred) |
| pyarrow / parquet-cpp | ✅ Correct |
### Expected behavior
Files written by `parquet-rs` should be readable by `parquet-mr` (the
reference Java implementation). Page indexes are optional metadata — their
presence should not prevent data from being read.
### Possible fixes
1. **Add an option to disable writing page indexes** (e.g.,
`WriterProperties::set_page_index_enabled(false)`) — immediate workaround for
users
2. **Fix the page index format** to be compatible with parquet-mr's reader
3. **Investigate if statistics in the page index are written incorrectly**
(e.g., null min/max causing parquet-mr to prune all pages)
### Environment
- parquet-rs version: 58.3.0
- Arrow-rs version: 58.3.0 (via OpenSearch composite engine)
- Spark: 3.5.x (Glue 4.0, EMR 7.0)
- Trino: Athena engine v3
- pyarrow: 7.0+ (reads correctly)
### Related
- [trinodb/trino#26058](https://github.com/trinodb/trino/issues/26058) —
Same symptom, fixed in Trino #26064 but not yet in AWS Athena
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
