[
https://issues.apache.org/jira/browse/BEAM-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319337#comment-16319337
]
ASF GitHub Bot commented on BEAM-3416:
--------------------------------------
chamikaramj closed pull request #4349: [BEAM-3416] Fixes unclosed files in
VcfSource when exception is thrown.
URL: https://github.com/apache/beam/pull/4349
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/io/vcfio.py
b/sdks/python/apache_beam/io/vcfio.py
index 80f4631e462..a45861ef82a 100644
--- a/sdks/python/apache_beam/io/vcfio.py
+++ b/sdks/python/apache_beam/io/vcfio.py
@@ -285,9 +285,12 @@ def __init__(self,
try:
self._vcf_reader = vcf.Reader(fsock=self._create_generator())
except SyntaxError as e:
- raise ValueError('An exception was raised when reading header from VCF
'
- 'file %s: %s' % (self._file_name,
- traceback.format_exc(e)))
+ # Throw the exception inside the generator to ensure file is properly
+ # closed (it's opened inside TextSource.read_records).
+ self._text_lines.throw(
+ ValueError('An exception was raised when reading header from VCF '
+ 'file %s: %s' % (self._file_name,
+ traceback.format_exc(e))))
def _store_header_lines(self, header_lines):
self._header_lines = header_lines
@@ -321,11 +324,14 @@ def next(self):
self._file_name, self._last_record, traceback.format_exc(e))
return MalformedVcfRecord(self._file_name, self._last_record)
- raise ValueError('An exception was raised when reading record from VCF
'
- 'file %s. Invalid record was %s: %s' % (
- self._file_name,
- self._last_record,
- traceback.format_exc(e)))
+ # Throw the exception inside the generator to ensure file is properly
+ # closed (it's opened inside TextSource.read_records).
+ self._text_lines.throw(
+ ValueError('An exception was raised when reading record from VCF '
+ 'file %s. Invalid record was %s: %s' % (
+ self._file_name,
+ self._last_record,
+ traceback.format_exc(e))))
def _convert_to_variant_record(self, record, infos, formats):
"""Converts the PyVCF record to a :class:`Variant` object.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> File is not properly close in VcfSource when exception is thrown
> ----------------------------------------------------------------
>
> Key: BEAM-3416
> URL: https://issues.apache.org/jira/browse/BEAM-3416
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Asha Rostamianfar
> Assignee: Asha Rostamianfar
> Priority: Minor
>
> Files are not properly closed when an exception is thrown in VcfSource
> (apache_beam/io/vcfio.py).
> This is because the file is opened within TextSource (using a 'with'
> statement), but the exception in VcfSource is thrown outside of that scope.
> This only causes an issue in Windows unit tests where files cannot be deleted
> if they're used by another process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
