[
https://issues.apache.org/jira/browse/BEAM-2732?focusedWorklogId=92231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92231
]
ASF GitHub Bot logged work on BEAM-2732:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Apr/18 20:27
Start Date: 18/Apr/18 20:27
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #4387: [BEAM-2732] Metrics
rely on statesampler state
URL: https://github.com/apache/beam/pull/4387#issuecomment-382518272
Monsterpaste:
```
======================================================================
ERROR: test_basics_with_type_check
(apache_beam.examples.cookbook.group_with_coder_test.GroupWithCoderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py",
line 53, in test_basics_with_type_check
'--output=%s.result' % temp_path])
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/cookbook/group_with_coder.py",
line 118, in run
| WriteToText(known_args.output))
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 522, in __init__
skip_header_lines=skip_header_lines)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 117, in __init__
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmpJcX3Kr*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
-------------------- >> begin captured logging << --------------------
root: INFO: Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test_basics_without_type_check
(apache_beam.examples.cookbook.group_with_coder_test.GroupWithCoderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py",
line 74, in test_basics_without_type_check
'--output=%s.result' % temp_path])
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/cookbook/group_with_coder.py",
line 118, in run
| WriteToText(known_args.output))
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 522, in __init__
skip_header_lines=skip_header_lines)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 117, in __init__
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmpisQlPp*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
-------------------- >> begin captured logging << --------------------
root: INFO: Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test_multiple_output_pardo
(apache_beam.examples.cookbook.multiple_output_pardo_test.MultipleOutputParDo)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py",
line 56, in test_multiple_output_pardo
'--output=%s' % result_prefix])
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py",
line 154, in run
lines = p | ReadFromText(known_args.input)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 522, in __init__
skip_header_lines=skip_header_lines)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 117, in __init__
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmpnmwepq*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
-------------------- >> begin captured logging << --------------------
root: INFO: Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test_basics
(apache_beam.examples.wordcount_debugging_test.WordCountTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/wordcount_debugging_test.py",
line 52, in test_basics
'--output=%s.result' % temp_path])
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/wordcount_debugging.py",
line 134, in run
| 'FilterText' >> beam.ParDo(FilterTextFn('Flourish|stomach')))
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 522, in __init__
skip_header_lines=skip_header_lines)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 117, in __init__
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmpSmPq3X*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
-------------------- >> begin captured logging << --------------------
root: INFO: Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test_basics
(apache_beam.examples.wordcount_minimal_test.WordCountMinimalTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/wordcount_minimal_test.py",
line 47, in test_basics
'--output=%s.result' % temp_path])
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/wordcount_minimal.py",
line 100, in run
lines = p | ReadFromText(known_args.input)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 522, in __init__
skip_header_lines=skip_header_lines)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 117, in __init__
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmprNRvNX*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
======================================================================
ERROR: test_basics (apache_beam.examples.wordcount_test.WordCountTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/wordcount_test.py",
line 46, in test_basics
'--output=%s.result' % temp_path])
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/examples/wordcount.py",
line 90, in run
lines = p | 'read' >> ReadFromText(known_args.input)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 522, in __init__
skip_header_lines=skip_header_lines)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/textio.py",
line 117, in __init__
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmpnAxeZ2*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
-------------------- >> begin captured logging << --------------------
root: INFO: Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test_sink_transform (apache_beam.io.avroio_test.TestAvro)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/avroio_test.py",
line 390, in test_sink_transform
readback = p | avroio.ReadFromAvro(path + '*') | beam.Map(json.dumps)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/avroio.py",
line 137, in __init__
self._source = _AvroSource(file_pattern, min_bundle_size,
validate=validate)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 119, in __init__
self._validate()
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/options/value_provider.py",
line 124, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filebasedsource.py",
line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystems.py",
line 166, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/pabloem/codes/global-sampler-metrics/sdks/python/apache_beam/io/filesystem.py",
line 600, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmpN4Vb7G*':
BeamIOError("List operation failed with exceptions {'/tmp': OSError(2, 'No such
file or directory')}",)}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 92231)
Time Spent: 12.5h (was: 12h 20m)
> State tracking in Python is inefficient and has duplicated code
> ---------------------------------------------------------------
>
> Key: BEAM-2732
> URL: https://issues.apache.org/jira/browse/BEAM-2732
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Pablo Estrada
> Assignee: Pablo Estrada
> Priority: Major
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> e.g logging and metrics keep state separately. State tracking should be
> unified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
