jorisvandenbossche commented on code in PR #14717:
URL: https://github.com/apache/arrow/pull/14717#discussion_r1037948892
##########
python/pyarrow/parquet/core.py:
##########
@@ -304,7 +308,15 @@ def __init__(self, source, *, metadata=None,
common_metadata=None,
read_dictionary=None, memory_map=False, buffer_size=0,
pre_buffer=False, coerce_int96_timestamp_unit=None,
decryption_properties=None, thrift_string_size_limit=None,
- thrift_container_size_limit=None):
+ thrift_container_size_limit=None, filesystem=None):
+
+ self._close_source = getattr(source, 'closed', True)
+
+ filesystem, source = _resolve_filesystem_and_path(source, filesystem)
Review Comment:
When passing a local path as string, this will now always return a
filesystem object. I think that means that the `memory_map` option will be
ignored (which is otherwise passed to `reader.open(..)`, and there passed to
`get_reader`.
So we might need to update `_resolve_filesystem_and_path` a bit to pass
through the memory_map option (a local filesystem can take that as a param, so
it can be passed through).
##########
python/pyarrow/tests/parquet/test_parquet_file.py:
##########
@@ -177,8 +178,12 @@ def
test_parquet_file_pass_directory_instead_of_file(tempdir):
path = tempdir / 'directory'
os.mkdir(str(path))
- with pytest.raises(IOError, match="Expected file path"):
+ msg = f"Cannot open for reading: path '{str(path)}' is a directory"
+ with pytest.raises(IOError) as exc:
pq.ParquetFile(path)
+ if exc.errisinstance(PermissionError) and sys.platform == 'win32':
+ return # Windows CI can get a PermissionError here.
Review Comment:
cc @pitrou do you have any idea why trying to open a directory on windows
gives a PermissionError instead of a normal IO error with the informative
message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
