DanielMorales9 commented on PR #32879:
URL: https://github.com/apache/beam/pull/32879#issuecomment-2426452525
Yep, makes sense! 👍
I checked once again and I can confirm that the manifest files are stored in
the manifest list along with their Partition Spec Id and the writing process
groups Data Files by their Partition Spec Id. We can merge ✅
Example Manifest list:
```{
"manifest_path":
"gs://<bucket>/<schema>/<table>/metadata/c4aa6dc6-5e0b-454c-b126-1d97d2f223f8-m1.avro",
"manifest_length": 8363467,
"partition_spec_id": 0,
"content": 0,
"sequence_number": 4814,
"min_sequence_number": 4093,
"added_snapshot_id": 481579842725173797,
"added_data_files_count": 0,
"existing_data_files_count": 24356,
"deleted_data_files_count": 0,
"added_rows_count": 0,
"existing_rows_count": 785677,
"deleted_rows_count": 0,
"partitions": {
"array": []
}
}
{
"manifest_path":
"gs://<bucket>/<schema>/<table>//metadata/9753c692-b48e-49bb-9c27-713c8a41842e-m1.avro",
"manifest_length": 8372685,
"partition_spec_id": 0,
"content": 0,
"sequence_number": 4093,
"min_sequence_number": 3557,
"added_snapshot_id": 1992827298894205835,
"added_data_files_count": 0,
"existing_data_files_count": 19878,
"deleted_data_files_count": 0,
"added_rows_count": 0,
"existing_rows_count": 1263244,
"deleted_rows_count": 0,
"partitions": {
"array": []
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
