Fokko commented on code in PR #8015:
URL: https://github.com/apache/iceberg/pull/8015#discussion_r1262601138
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -521,6 +521,10 @@ def _construct_fragment(fs: FileSystem, data_file:
DataFile, file_format_kwargs:
return _get_file_format(data_file.file_format,
**file_format_kwargs).make_fragment(path, fs)
+def _starmap_read_deletes(args: Tuple[FileSystem, DataFile]) -> Dict[str,
pa.ChunkedArray]:
Review Comment:
For me, this passthrough function does not add a lot of value. WDYT of:
```python
def _read_deletes(args: Tuple[FileSystem, DataFile]) -> Dict[str,
pa.ChunkedArray]:
fs, data_file = args
```
##########
python/pyiceberg/table/__init__.py:
##########
@@ -773,11 +779,11 @@ def plan_files(self) -> Iterable[FileScanTask]:
data_entries: List[ManifestEntry] = []
positional_delete_entries = SortedList(key=lambda entry:
entry.data_sequence_number or INITIAL_SEQUENCE_NUMBER)
- with ThreadPool() as pool:
+ with DynamicExecutor() as executor:
Review Comment:
Yes, this works for me. Can you also add this to the docs under
`python/mkdocs/`?
##########
python/pyiceberg/table/__init__.py:
##########
@@ -639,6 +639,12 @@ def __init__(
self.length = length or data_file.file_size_in_bytes
+def _starmap_open_manifest(
Review Comment:
Same here, maybe better to let `_open_manifest` accept a single tuple
argument, and then we can directly expand those.
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -725,6 +729,22 @@ def primitive(self, primitive: pa.DataType) -> IcebergType:
raise TypeError(f"Unsupported type: {primitive}")
+def _starmap_task_to_table(
Review Comment:
Same here, maybe better to let `_task_to_table` accept a single tuple
argument, and then we can directly expand those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
