alamb commented on code in PR #118:
URL: https://github.com/apache/parquet-testing/pull/118#discussion_r3707244134
##########
data/bson.parquet:
##########
Review Comment:
I verified the contents of this with datafusion-cli:
```sql
$ datafusion-cli -c "SELECT * from 'data/json.parquet'"
DataFusion CLI v54.1.0
+------------------+
| json_field |
+------------------+
| {"a":1} |
| {"a":1,"b":null} |
| [1,null,3] |
| NULL |
+------------------+
4 row(s) fetched.
Elapsed 0.039 seconds.
```
##########
data/json.parquet:
##########
Review Comment:
Contents look good:
```sql
datafusion-cli -c "SELECT * from 'data/bson.parquet'"
DataFusion CLI v54.1.0
+--------------------------------+
| bson_field |
+--------------------------------+
| 0c0000001061000100000000 |
| 0f000000106100010000000a620000 |
| NULL |
+--------------------------------+
3 row(s) fetched.
```
I decoded into bson and it matches the contents:
```python
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ uv run --with
pymongo --with pyarrow python3 -c "
import bson, pyarrow.parquet as pq
t = pq.read_table('data/bson.parquet')
for v in t.column('bson_field'):
print('NULL' if not v.is_valid else bson.decode(v.as_py()))
"
{'a': 1}
{'a': 1, 'b': None}
NULL
```
As does metadata
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$
~/Software/arrow-rs/target/release/parquet-schema data/bson.parquet
Metadata for file: data/bson.parquet
version: 1
num of rows: 3
created by: parquet-mr version 1.18.0-SNAPSHOT (build
1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message bson_fixture {
OPTIONAL BYTE_ARRAY bson_field (BSON);
}
```
##########
data/bson.parquet:
##########
Review Comment:
Metadata is good too
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$
~/Software/arrow-rs/target/release/parquet-schema data/json.parquet
Metadata for file: data/json.parquet
version: 1
num of rows: 4
created by: parquet-mr version 1.18.0-SNAPSHOT (build
1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message json_fixture {
OPTIONAL BYTE_ARRAY json_field (JSON);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
