[
https://issues.apache.org/jira/browse/BEAM-6821?focusedWorklogId=214386&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214386
]
ASF GitHub Bot logged work on BEAM-6821:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Mar/19 08:33
Start Date: 17/Mar/19 08:33
Worklog Time Spent: 10m
Work Description: gkovelman commented on issue #8054: [BEAM-6821]
FileBasedSink improper paths
URL: https://github.com/apache/beam/pull/8054#issuecomment-473632585
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 214386)
Time Spent: 1h (was: 50m)
> FileBasedSink is not creating file paths according to target filesystem
> -----------------------------------------------------------------------
>
> Key: BEAM-6821
> URL: https://issues.apache.org/jira/browse/BEAM-6821
> Project: Beam
> Issue Type: Bug
> Components: io-java-text
> Affects Versions: 2.11.0
> Environment: Windows 10
> Reporter: Gregory Kovelman
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> File path generated in _open_writer_ method is not according to target
> filesystem, because
> os.path.join is used and not FileSystems.join.
> apache_beam\io\filebasedsink.py extract:
>
> {code:java}
> def _create_temp_dir(self, file_path_prefix):
> base_path, last_component = FileSystems.split(file_path_prefix)
> if not last_component:
> # Trying to re-split the base_path to check if it's a root.
> new_base_path, _ = FileSystems.split(base_path)
> if base_path == new_base_path:
> raise ValueError('Cannot create a temporary directory for root path '
> 'prefix %s. Please specify a file path prefix with '
> 'at least two components.' % file_path_prefix)
> path_components = [base_path,
> 'beam-temp-' + last_component + '-' + uuid.uuid1().hex]
> return FileSystems.join(*path_components)
> @check_accessible(['file_path_prefix', 'file_name_suffix'])
> def open_writer(self, init_result, uid):
> # A proper suffix is needed for AUTO compression detection.
> # We also ensure there will be no collisions with uid and a
> # (possibly unsharded) file_path_prefix and a (possibly empty)
> # file_name_suffix.
> file_path_prefix = self.file_path_prefix.get()
> file_name_suffix = self.file_name_suffix.get()
> suffix = (
> '.' + os.path.basename(file_path_prefix) + file_name_suffix)
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> {code}
>
>
> This created incompatibilities between, for example, Windows and GCS.
> Expected: gs://bucket/beam-temp-result-uuid\\uid.result
> Actual: gs://bucket/beam-temp-result-uuid/uid.result
> Replacing os.path.join with FileSystems.join fixes the issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
