Wes McKinney created ARROW-2306:
-----------------------------------
Summary: [Python] HDFS test failures
Key: ARROW-2306
URL: https://issues.apache.org/jira/browse/ARROW-2306
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Wes McKinney
Assignee: Wes McKinney
Fix For: 0.9.0
These weren't caught because we aren't running the HDFS tests in Travis CI
{code}
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_write_to_dataset_no_partitions
FAILED
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
self = <pyarrow.tests.test_hdfs.TestLibHdfs
testMethod=test_write_to_dataset_no_partitions>
@test_parquet.parquet
def test_write_to_dataset_no_partitions(self):
tmpdir = pjoin(self.tmp_path, 'write-no_partitions-' + guid())
self.hdfs.mkdir(tmpdir)
test_parquet._test_write_to_dataset_no_partitions(
> tmpdir, filesystem=self.hdfs)
pyarrow/tests/test_hdfs.py:367:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyarrow/tests/test_parquet.py:1475: in _test_write_to_dataset_no_partitions
filesystem=filesystem)
pyarrow/parquet.py:1059: in write_to_dataset
_mkdir_if_not_exists(fs, root_path)
pyarrow/parquet.py:1006: in _mkdir_if_not_exists
if fs._isfilestore() and not fs.exists(path):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyarrow.hdfs.HadoopFileSystem object at 0x7f09e87a4c48>
def _isfilestore(self):
"""
Returns True if this FileSystem is a unix-style file store with
directories.
"""
> raise NotImplementedError
E NotImplementedError
pyarrow/filesystem.py:143: NotImplementedError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/code/arrow/python/pyarrow/filesystem.py(143)_isfilestore()
-> raise NotImplementedError
(Pdb) c
pyarrow/tests/test_hdfs.py::TestLibHdfs::test_write_to_dataset_with_partitions
FAILED
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
self = <pyarrow.tests.test_hdfs.TestLibHdfs
testMethod=test_write_to_dataset_with_partitions>
@test_parquet.parquet
def test_write_to_dataset_with_partitions(self):
tmpdir = pjoin(self.tmp_path, 'write-partitions-' + guid())
self.hdfs.mkdir(tmpdir)
test_parquet._test_write_to_dataset_with_partitions(
> tmpdir, filesystem=self.hdfs)
pyarrow/tests/test_hdfs.py:360:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyarrow/tests/test_parquet.py:1433: in _test_write_to_dataset_with_partitions
filesystem=filesystem)
pyarrow/parquet.py:1059: in write_to_dataset
_mkdir_if_not_exists(fs, root_path)
pyarrow/parquet.py:1006: in _mkdir_if_not_exists
if fs._isfilestore() and not fs.exists(path):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyarrow.hdfs.HadoopFileSystem object at 0x7f09e87a4c48>
def _isfilestore(self):
"""
Returns True if this FileSystem is a unix-style file store with
directories.
"""
> raise NotImplementedError
E NotImplementedError
pyarrow/filesystem.py:143: NotImplementedError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/code/arrow/python/pyarrow/filesystem.py(143)_isfilestore()
-> raise NotImplementedError
(Pdb) c
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
