[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[
https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=82038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82038
]
ASF GitHub Bot logged work on BEAM-3759:
Author: ASF GitHub Bot
Created on: 19/Mar/18 20:57
Start Date: 19/Mar/18 20:57
Worklog Time Spent: 10m
Work Description: robertwb closed pull request #4763: [BEAM-3759] Add
support for PaneInfo in WindowedValues
URL: https://github.com/apache/beam/pull/4763
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/coders/coder_impl.pxd
b/sdks/python/apache_beam/coders/coder_impl.pxd
index 8af394b6686..98dd508556a 100644
--- a/sdks/python/apache_beam/coders/coder_impl.pxd
+++ b/sdks/python/apache_beam/coders/coder_impl.pxd
@@ -132,11 +132,16 @@ cdef class IterableCoderImpl(SequenceCoderImpl):
pass
+cdef class PaneInfoCoderImpl(StreamCoderImpl):
+ cdef int _choose_encoding(self, value)
+
+
cdef class WindowedValueCoderImpl(StreamCoderImpl):
"""A coder for windowed values."""
cdef CoderImpl _value_coder
cdef CoderImpl _timestamp_coder
cdef CoderImpl _windows_coder
+ cdef CoderImpl _pane_info_coder
@cython.locals(c=CoderImpl)
cpdef get_estimated_size_and_observables(self, value, bint nested=?)
diff --git a/sdks/python/apache_beam/coders/coder_impl.py
b/sdks/python/apache_beam/coders/coder_impl.py
index d47357df8c9..cc7ed87c3ad 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -672,6 +672,82 @@ def _construct_from_sequence(self, components):
return components
+class PaneInfoEncoding(object):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ Encoding used to describe a PaneInfo descriptor. A PaneInfo descriptor
+ can be encoded in three different ways: with a single byte (FIRST), with a
+ single byte followed by a varint describing a single index (ONE_INDEX) or
+ with a single byte followed by two varints describing two separate indices:
+ the index and nonspeculative index.
+ """
+
+ FIRST = 0
+ ONE_INDEX = 1
+ TWO_INDICES = 2
+
+
+class PaneInfoCoderImpl(StreamCoderImpl):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ Coder for a PaneInfo descriptor."""
+
+ def _choose_encoding(self, value):
+if ((value.index == 0 and value.nonspeculative_index == 0) or
+value.timing == windowed_value.PaneInfoTiming.UNKNOWN):
+ return PaneInfoEncoding.FIRST
+elif (value.index == value.nonspeculative_index or
+ value.timing == windowed_value.PaneInfoTiming.EARLY):
+ return PaneInfoEncoding.ONE_INDEX
+else:
+ return PaneInfoEncoding.TWO_INDICES
+
+ def encode_to_stream(self, value, out, nested):
+encoding_type = self._choose_encoding(value)
+out.write_byte(value.encoded_byte | (encoding_type << 4))
+if encoding_type == PaneInfoEncoding.FIRST:
+ return
+elif encoding_type == PaneInfoEncoding.ONE_INDEX:
+ out.write_var_int64(value.index)
+elif encoding_type == PaneInfoEncoding.TWO_INDICES:
+ out.write_var_int64(value.index)
+ out.write_var_int64(value.nonspeculative_index)
+else:
+ raise NotImplementedError('Invalid PaneInfoEncoding: %s' % encoding_type)
+
+ def decode_from_stream(self, in_stream, nested):
+encoded_first_byte = in_stream.read_byte()
+base = windowed_value._BYTE_TO_PANE_INFO[encoded_first_byte & 0xF]
+assert base is not None
+encoding_type = encoded_first_byte >> 4
+if encoding_type == PaneInfoEncoding.FIRST:
+ return base
+elif encoding_type == PaneInfoEncoding.ONE_INDEX:
+ index = in_stream.read_var_int64()
+ if base.timing == windowed_value.PaneInfoTiming.EARLY:
+nonspeculative_index = -1
+ else:
+nonspeculative_index = index
+elif encoding_type == PaneInfoEncoding.TWO_INDICES:
+ index = in_stream.read_var_int64()
+ nonspeculative_index = in_stream.read_var_int64()
+else:
+ raise NotImplementedError('Invalid PaneInfoEncoding: %s' % encoding_type)
+return windowed_value.PaneInfo(
+base.is_first, base.is_last, base.timing, index, nonspeculative_index)
+
+ def estimate_size(self, value, nested=False):
+"""Estimates the encoded size of the given value, in bytes."""
+size = 1
+encoding_type = self._choose_encoding(value)
+if encoding_type == PaneInfoEncoding.ONE_INDEX:
+ size += get_varint_size(value.index)
+elif encoding_type == PaneInfoEncoding.TWO_INDICES:
+ size += get_varint_size(value.index)
+ size += get_varint_size(value.nonspeculative_index)
+return size
+
+
class
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=82037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82037 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 20:57 Start Date: 19/Mar/18 20:57 Worklog Time Spent: 10m Work Description: robertwb commented on issue #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#issuecomment-374373647 Totally within the margin of error. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 82037) Time Spent: 3.5h (was: 3h 20m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=82034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82034 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 20:50 Start Date: 19/Mar/18 20:50 Worklog Time Spent: 10m Work Description: charlesccychen commented on issue #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#issuecomment-374370997 Thanks. The benchmarks seem very comparable. Without this change (at HEAD^): ``` [(1, 1.0581729412078857)] [(1, 1.0581729412078857), (1001, 1.0961029529571533)] [(1, 1.0581729412078857), (1001, 1.0961029529571533), (2001, 1.304905891418457)] [(1, 1.0581729412078857), (1001, 1.0961029529571533), (2001, 1.304905891418457), (3001, 1.3804380893707275)] [(1, 1.0581729412078857), (1001, 1.0961029529571533), (2001, 1.304905891418457), (3001, 1.3804380893707275), (4001, 1.6390268802642822)] Fixed cost 1.00637614384 Per-element 1.44604301453e-06 R^2 0.94447668637 ``` With this change: ``` [(1, 1.0614569187164307)] [(1, 1.0614569187164307), (1001, 1.1248741149902344)] [(1, 1.0614569187164307), (1001, 1.1248741149902344), (2001, 1.315816879272461)] [(1, 1.0614569187164307), (1001, 1.1248741149902344), (2001, 1.315816879272461), (3001, 1.3426880836486816)] [(1, 1.0614569187164307), (1001, 1.1248741149902344), (2001, 1.315816879272461), (3001, 1.3426880836486816), (4001, 1.6559441089630127)] Fixed cost 1.01865767245 Per-element 1.40678834915e-06 R^2 0.914786773244 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 82034) Time Spent: 3h 20m (was: 3h 10m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81741 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 06:43 Start Date: 19/Mar/18 06:43 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#discussion_r175341742 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -30,8 +30,6 @@ from types import NoneType -import six - from apache_beam.coders import observable Review comment: > **robertwb** wrote: > Or perhaps just remove this, as the comment below is sufficient. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 81741) Time Spent: 2.5h (was: 2h 20m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[
https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81742
]
ASF GitHub Bot logged work on BEAM-3759:
Author: ASF GitHub Bot
Created on: 19/Mar/18 06:43
Start Date: 19/Mar/18 06:43
Worklog Time Spent: 10m
Work Description: charlesccychen commented on a change in pull request
#4763: [BEAM-3759] Add support for PaneInfo in WindowedValues
URL: https://github.com/apache/beam/pull/4763#discussion_r175341749
##
File path: sdks/python/apache_beam/utils/windowed_value.py
##
@@ -32,6 +32,107 @@
from apache_beam.utils.timestamp import Timestamp
+class PaneInfoTiming(object):
+ """The timing of a PaneInfo."""
+
+ EARLY = 0
+ ON_TIME = 1
+ LATE = 2
+ UNKNOWN = 3
+
+
+class PaneInfo(object):
+ """Describes the trigger firing information for a given WindowedValue."""
+
+ def __init__(self, is_first, is_last, timing, index, nonspeculative_index):
+self._is_first = is_first
+self._is_last = is_last
+self._timing = timing
+self._index = index
+self._nonspeculative_index = nonspeculative_index
+self._encoded_byte = self._get_encoded_byte()
+
+ def _get_encoded_byte(self):
+byte = 0
+if self.is_first:
+ byte |= 1
+if self.is_last:
+ byte |= 2
+byte |= self.timing << 2
+return byte
+
+ @staticmethod
+ def from_encoded_byte(encoded_byte):
+assert encoded_byte in _BYTE_TO_PANE_INFO
+return _BYTE_TO_PANE_INFO[encoded_byte]
+
+ # Because common PaneInfo objects are cached, it is important that the value
+ # is immutable. We therefore explicitly enforce this here with read-only
+ # properties.
+
+ @property
+ def is_first(self):
+return self._is_first
+
+ @property
+ def is_last(self):
+return self._is_last
+
+ @property
+ def timing(self):
+return self._timing
+
+ @property
+ def index(self):
+return self._index
+
+ @property
+ def nonspeculative_index(self):
+return self._nonspeculative_index
+
+ @property
+ def encoded_byte(self):
+return self._encoded_byte
+
+ def __repr__(self):
+return ('PaneInfo(first: %r, last: %r, timing: %s, index: %d, '
+'nonspeculative_index: %d)') % (self.is_first, self.is_last,
+self.timing, self.index,
+self.nonspeculative_index)
+
+ def __eq__(self, other):
+if self is other:
+ return True
+return (self.is_first == other.is_first and
+self.is_last == other.is_last and
+self.timing == other.timing and
+self.index == other.index and
+self.nonspeculative_index == other.nonspeculative_index)
+
+
+def _construct_pane_info_map():
+ result = {}
Review comment:
> **robertwb** wrote:
> Make this a list, not a map.
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 81742)
Time Spent: 2h 40m (was: 2.5h)
> Add support for PaneInfo descriptor in Python SDK
> -
>
> Key: BEAM-3759
> URL: https://issues.apache.org/jira/browse/BEAM-3759
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.3.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The PaneInfo descriptor allows a user to determine which particular
> triggering emitted a value. This allows the user to differentiate between
> speculative (early), on-time (at end of window) and late value emissions
> coming out of a GroupByKey. We should add support for this feature in the
> Python SDK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81739 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 06:43 Start Date: 19/Mar/18 06:43 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#discussion_r175341748 ## File path: sdks/python/apache_beam/utils/windowed_value.py ## @@ -104,11 +211,12 @@ def __reduce__(self): # TODO(robertwb): Move this to a static method. -def create(value, timestamp_micros, windows): +def create(value, timestamp_micros, windows, pane_info=None): Review comment: > **robertwb** wrote: > Just let the default value be PANE_INFO_UNKNOWN. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 81739) Time Spent: 2h 10m (was: 2h) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[
https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81744
]
ASF GitHub Bot logged work on BEAM-3759:
Author: ASF GitHub Bot
Created on: 19/Mar/18 06:43
Start Date: 19/Mar/18 06:43
Worklog Time Spent: 10m
Work Description: charlesccychen commented on a change in pull request
#4763: [BEAM-3759] Add support for PaneInfo in WindowedValues
URL: https://github.com/apache/beam/pull/4763#discussion_r175341743
##
File path: sdks/python/apache_beam/coders/coders.py
##
@@ -878,6 +877,27 @@ def __hash__(self):
common_urns.INTERVAL_WINDOW_CODER, IntervalWindowCoder)
+class PaneInfoCoder(FastCoder):
+ """Coder for an PaneInfo descriptor."""
+
+ def _create_impl(self):
+return coder_impl.PaneInfoCoderImpl()
+
+ def is_deterministic(self):
+return True
+
+ def as_cloud_object(self):
+raise NotImplementedError(
+'The PaneInfoCoder should not be explicitly constructed in a pipeline '
+'graph.')
+
+ def __eq__(self, other):
+return type(self) == type(other)
+
+ def __hash__(self):
+return hash(type(self))
+
+
Review comment:
> **robertwb** wrote:
> Rather than introducing a PaneInfoCoder, just instantiate the
PaneInfoCoderImpl in WindowedValueCoderImpl's constructor. (There's a TODO to
remove the parameterizablility of the TimestampCoder as well.) Actually, we
could consider just making {en,de}code_pane_info methods on WindowedValueCoder
itself.
I am going to get rid of PaneInfoCoder but keep the PaneInfoCoderImpl,
because it's more natural to keep it separate, as it would clutter the
WindowedValueCoderImpl with encode/decode_paneinfo_to_stream and
estimate_paneinfo_size.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 81744)
Time Spent: 3h (was: 2h 50m)
> Add support for PaneInfo descriptor in Python SDK
> -
>
> Key: BEAM-3759
> URL: https://issues.apache.org/jira/browse/BEAM-3759
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.3.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The PaneInfo descriptor allows a user to determine which particular
> triggering emitted a value. This allows the user to differentiate between
> speculative (early), on-time (at end of window) and late value emissions
> coming out of a GroupByKey. We should add support for this feature in the
> Python SDK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81740 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 06:43 Start Date: 19/Mar/18 06:43 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#discussion_r175341741 ## File path: sdks/python/apache_beam/utils/windowed_value.py ## @@ -90,7 +196,8 @@ def __cmp__(left, right): # pylint: disable=no-self-argument def _typed_eq(left, right): return (left.timestamp_micros == right.timestamp_micros and left.value == right.value -and left.windows == right.windows) +and left.windows == right.windows +and left.pane_info == right.pane_info) def with_value(self, new_value): """Creates a new WindowedValue with the same timestamps and windows as this. Review comment: > **robertwb** wrote: > Fix with_value to propagate the pane info (and test). Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 81740) Time Spent: 2h 20m (was: 2h 10m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81745 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 06:43 Start Date: 19/Mar/18 06:43 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#discussion_r175341745 ## File path: sdks/python/apache_beam/utils/windowed_value.py ## @@ -104,11 +211,12 @@ def __reduce__(self): Review comment: > **robertwb** wrote: > Fix `__reduce__` to rememeber the PaneInfo (and test). Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 81745) Time Spent: 3h 10m (was: 3h) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81743 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 19/Mar/18 06:43 Start Date: 19/Mar/18 06:43 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#discussion_r175341744 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -689,11 +766,13 @@ def _from_normal_time(self, value): Review comment: > **robertwb** wrote: > You can now remove this TODO. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 81743) Time Spent: 2h 50m (was: 2h 40m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81453 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 17/Mar/18 00:19 Start Date: 17/Mar/18 00:19 Worklog Time Spent: 10m Work Description: aaltay commented on issue #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#issuecomment-373877868 @charlesccychen any updates on this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 81453) Time Spent: 2h (was: 1h 50m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=80893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-80893 ] ASF GitHub Bot logged work on BEAM-3759: Author: ASF GitHub Bot Created on: 15/Mar/18 17:24 Start Date: 15/Mar/18 17:24 Worklog Time Spent: 10m Work Description: robertwb commented on a change in pull request #4763: [BEAM-3759] Add support for PaneInfo in WindowedValues URL: https://github.com/apache/beam/pull/4763#discussion_r173303136 ## File path: sdks/python/apache_beam/utils/windowed_value.py ## @@ -104,11 +211,12 @@ def __reduce__(self): Review comment: Fix `__reduce__` to rememeber the PaneInfo (and test). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 80893) Time Spent: 1h 50m (was: 1h 40m) > Add support for PaneInfo descriptor in Python SDK > - > > Key: BEAM-3759 > URL: https://issues.apache.org/jira/browse/BEAM-3759 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.3.0 >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > The PaneInfo descriptor allows a user to determine which particular > triggering emitted a value. This allows the user to differentiate between > speculative (early), on-time (at end of window) and late value emissions > coming out of a GroupByKey. We should add support for this feature in the > Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
