+1 For uncompressed size for the field. However, it's a bit-tricky here.
I've
implement a similar size-hint in our system, here are some problems I met:
1. Null variables. In Arrow Array, null-value should occupy some place, but
field-raw size cannot represent that value.
2. Size of FLBA/ByteArray. It's size should be variable-size-summary or
variable-size-summary + sizeof(ByteArray) * value-count
3. Some times Arrow data is not equal to Parquet data, like Decimal stored
as int32 or int64.
Hope that helps.
- [DISCUSS] Add a Plain Encoding Size Bytes to Parquet Metad... Micah Kornfield
- Re: [DISCUSS] Add a Plain Encoding Size Bytes to Parq... Will Jones
- Re: [DISCUSS] Add a Plain Encoding Size Bytes to ... Gang Wu
- Re: [DISCUSS] Add a Plain Encoding Size Bytes... Micah Kornfield
- Re: [DISCUSS] Add a Plain Encoding Size B... Micah Kornfield
- Re: [DISCUSS] Add a Plain Encoding S... Micah Kornfield
- RE: Re: [DISCUSS] Add a Plain Encoding Size Bytes to ... wish maple
- Re: Re: [DISCUSS] Add a Plain Encoding Size Bytes... Micah Kornfield
- RE: [DISCUSS] Add a Plain Encoding Size Bytes to Parq... wish maple
- RE: [DISCUSS] Add a Plain Encoding Size Bytes to Parq... wish maple
- Re: [DISCUSS] Add a Plain Encoding Size Bytes to ... Micah Kornfield
