kevinjqliu commented on code in PR #3652:
URL: https://github.com/apache/iceberg-rust/pull/3652#discussion_r4575403974
##########
bindings/python/src/manifest.rs:
##########
@@ -226,11 +233,91 @@ pub fn read_manifest_list(bs: &[u8]) ->
PyResult<PyManifestList> {
})
}
+/// Reads the entries of a manifest as a `pyarrow.RecordBatchReader`.
+///
+/// The other arguments come from the manifest's entry in the manifest list,
+/// which the entries inherit snapshot IDs, sequence numbers, and first row IDs
+/// from. Raises `ValueError` for an invalid manifest header. An error while
+/// reading raises `pyarrow.ArrowInvalid`, which subclasses `ValueError`. The
+/// reader holds a reference to `bs` until it's closed or collected, and the
+/// next call into `pyiceberg_core` releases it.
+#[pyfunction]
+#[pyo3(signature = (
+ bs,
+ *,
+ manifest_path,
+ content,
+ sequence_number,
+ added_snapshot_id,
+ first_row_id = None,
+))]
+pub fn read_manifest_entries_as_arrow(
+ py: Python<'_>,
+ bs: PyBackedBytes,
+ manifest_path: String,
+ content: i32,
+ sequence_number: i64,
+ added_snapshot_id: i64,
+ first_row_id: Option<u64>,
+) -> PyResult<Bound<'_, PyAny>> {
+ // Inheritance reads only the fields set from the arguments.
Review Comment:
this feels a bit awkward but i think its the best we can do right now
without major refactor
##########
bindings/python/src/manifest.rs:
##########
@@ -226,11 +233,91 @@ pub fn read_manifest_list(bs: &[u8]) ->
PyResult<PyManifestList> {
})
}
+/// Reads the entries of a manifest as a `pyarrow.RecordBatchReader`.
+///
+/// The other arguments come from the manifest's entry in the manifest list,
+/// which the entries inherit snapshot IDs, sequence numbers, and first row IDs
+/// from. Raises `ValueError` for an invalid manifest header. An error while
+/// reading raises `pyarrow.ArrowInvalid`, which subclasses `ValueError`. The
+/// reader holds a reference to `bs` until it's closed or collected, and the
+/// next call into `pyiceberg_core` releases it.
+#[pyfunction]
+#[pyo3(signature = (
+ bs,
+ *,
+ manifest_path,
+ content,
+ sequence_number,
+ added_snapshot_id,
+ first_row_id = None,
+))]
+pub fn read_manifest_entries_as_arrow(
Review Comment:
Nit: `_as_arrow` is ambiguous: PyIceberg uses `to_arrow()` for a `pa.Table`
and `to_arrow_batch_reader()` for a `RecordBatchReader`. #3648 also requested
tables.
Consider either:
- Rename to `read_manifest_entries_as_arrow_reader` and
`read_manifest_list_as_arrow_reader`.
- Move them into `pyiceberg_core.manifest.arrow` as `read_manifest_entries`
and `read_manifest_list`, mirroring Rust's `iceberg::arrow` module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]