jorisvandenbossche commented on code in PR #13821:
URL: https://github.com/apache/arrow/pull/13821#discussion_r941549225
##########
python/pyarrow/parquet/core.py:
##########
@@ -1790,6 +1803,10 @@ def __init__(self, path_or_paths, filesystem=None,
schema=None,
if validate_schema:
self.validate_schemas()
+ def close(self):
Review Comment:
Let's make this a private method, to not add methods to ParquetDataset that
we don't intend to keep long term (that don't exist for _ParquetDatasetV2)
##########
python/pyarrow/_parquet.pyx:
##########
@@ -1222,7 +1223,9 @@ cdef class ParquetReader(_Weakrefable):
self.source = source
- get_reader(source, use_memory_map, &rd_handle)
+ nf = get_native_file(source, use_memory_map)
+ (&rd_handle)[0] =
<shared_ptr[CRandomAccessFile]>nf.get_random_access_file()
+ self.nf = nf
Review Comment:
Instead of basically doing here what `get_reader` does under the hood, could
we modify `get_reader` to actually return the NativeFile object? (in other
cases where it is being used, this return value will just be discarded)
In that case we can here do something like:
```
self.nf = get_reader(source, use_memory_map, &rd_handle)
```
##########
python/pyarrow/parquet/core.py:
##########
@@ -375,6 +381,13 @@ def num_row_groups(self):
"""
return self.reader.num_row_groups
+ def close(self):
+ self.reader.close()
+
+ @property
+ def closed(self) -> bool:
+ return self.reader.closed
Review Comment:
Does this property exist on ParquetReader?
(this property also doesn't seem to be tested)
##########
python/pyarrow/parquet/core.py:
##########
@@ -1790,6 +1803,10 @@ def __init__(self, path_or_paths, filesystem=None,
schema=None,
if validate_schema:
self.validate_schemas()
+ def close(self):
Review Comment:
Actually, we also don't really need this method ourselves, since the read
method here calls `piece.read()`, and that is updated to close the handle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
