tschaub opened a new issue, #37814:
URL: https://github.com/apache/arrow/issues/37814
### Describe the bug, including details regarding any error messages,
version, and platform.
I'm using `pqarrow.FromParquet()` and `pqarrow.ToParquet()` to transform a
Parquet schema into an Arrow schema and back. It may be that I can avoid this,
but I'm starting with a Parquet schema and want to make use of the
`pqarrow.FileWriter`, so it seems like I need to work with an Arrow schema.
This is likely user error, but I was surprised to see that repetition
information was lost when going from a Parquet schema to an Arrow schema and
back. Specifically, repeated fields in Parquet become required lists.
Here is an example test that tries to round trip a schema with a single
repeated bool field:
```go
func TestRoundTripSchema(t *testing.T) {
fields := schema.FieldList{
schema.NewBooleanNode("bools", parquet.Repetitions.Repeated,
-1),
}
root, err := schema.NewGroupNode("root", parquet.Repetitions.Required,
fields, -1)
require.NoError(t, err)
parquetSchema1 := schema.NewSchema(root)
arrowSchema, err := pqarrow.FromParquet(parquetSchema1, nil, nil)
require.NoError(t, err)
writerProps := parquet.NewWriterProperties(
parquet.WithRootName("root"),
parquet.WithRootRepetition(parquet.Repetitions.Required),
)
parquetSchema2, err := pqarrow.ToParquet(arrowSchema, writerProps,
pqarrow.DefaultWriterProps())
require.NoError(t, err)
assert.True(t, parquetSchema1.Equals(parquetSchema2))
assert.Equal(t, parquetSchema1.String(), parquetSchema2.String())
}
```
The difference in the (string representation of the) schemas looks like this:
```diff
--- Expected
+++ Actual
@@ -1,3 +1,7 @@
required group field_id=-1 root {
- repeated boolean field_id=-1 bools;
+ required group field_id=-1 bools (List) {
+ repeated group field_id=-1 list {
+ optional boolean field_id=-1 element;
+ }
+ }
}
```
I see that there is some special meaning to the the
`metadata.KeyValueMetadata` pairs passed to
[`pqarrow.FromParquet()`](https://pkg.go.dev/github.com/apache/arrow/go/[email protected]/parquet/pqarrow#FromParquet).
Should I be passing information about whether to use lists or repeated values
there? Or is there a
[`parquet.WriterProperty`](https://pkg.go.dev/github.com/apache/arrow/go/[email protected]/parquet#WriterProperty)
that I should be passing along to
[`pqarrow.ToParquet()`](https://pkg.go.dev/github.com/apache/arrow/go/[email protected]/parquet/pqarrow#ToParquet)?
Thanks for any suggestions on how to retain the original repetition when
round-tripping a Parquet schema through an Arrow schema.
### Component(s)
Go, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
