4/27/2021
Attendees: Xinli Shang, Gábor Szádovszky, Gidon Gershinsky, Xin Dong, Ryan
Blue, Q. Xie
1.
Parquet 1.12.0 post release issues:
1.
Parquet-2027 - Page offset issue. Fixed.
2.
Parquet-2026 - Not allowing empty file anymore
1.
What are the data of empty file looks like?
1.
They are 0 rows files.
2.
What are they used?
1.
In some cases, they could be used as a placeholder.
3.
The previous release didn’t make it clear as a supported feature.
If we allow this in the future, we should document this.
Gabor will chat
with the owner if they can fix and contribute.
2.
INT-96 issue: Is it possible to fix it? For the read path, we already
have a configuration. So far it seems not easy for us to fix as it could
introduce several issues.
1.
Gabor will open a Jira ticket and investigate a little more to get a
sense of how complex and how risky it is. Once we have that
information, we
can discuss it next time.
3.
Provider-aware compression codec - PARQUET-1886
1.
If we can make it independent of Hadoop API, it could be better.
Let’s have a design for that. Make it configurable instead of implicitly
looking up because that could introduce runtime error.
4.
Iceberg related discussion
1.
Testing is going on with Iceberg for Parquet 1.12.0.
5.
Null data masking
1.
Null masking implemented in Parquet and it doesn't need to re-write
the data. It can be contributed to open source.
---
Xinli Shang | Tech Lead Manager in Uber Data Infra
