[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=372777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372777 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 16/Jan/20 04:37 Start Date: 16/Jan/20 04:37 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372777) Time Spent: 14h (was: 13h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 14h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=372764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372764 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 16/Jan/20 03:44 Start Date: 16/Jan/20 03:44 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-574968438 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372764) Time Spent: 13h 40m (was: 13.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 13h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=372765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372765 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 16/Jan/20 03:44 Start Date: 16/Jan/20 03:44 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-574968497 kicked off the tests since there have been a good number of changes recently This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372765) Time Spent: 13h 50m (was: 13h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 13h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=371974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371974 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 14/Jan/20 23:17 Start Date: 14/Jan/20 23:17 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-574419270 Sorry that I haven't been able to return to this in such a long while. I'll try to review it tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 371974) Time Spent: 13.5h (was: 13h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 13.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=371364=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371364 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 14/Jan/20 05:50 Start Date: 14/Jan/20 05:50 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-574013755 R: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 371364) Time Spent: 13h 20m (was: 13h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 13h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364905 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 07:39 Start Date: 31/Dec/19 07:39 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-569882002 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364905) Time Spent: 13h 10m (was: 13h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 13h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364868
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 31/Dec/19 05:40
Start Date: 31/Dec/19 05:40
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10464:
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362151892
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
// (Required) The local name of this timer for the PTransform that
references it.
string local_name = 2;
}
+
+ // Settings that decide the coder type of wire coder.
+ message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
Review comment:
Do you think we should add the WireCoderSetting property for each input and
output edge explicitly in the proto?
I was thinking the setting is not a necessary properties for each edge so
added the wire_coder_settings as an ExecutableStage level property. The
wire_coder_settings can be an empty list.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 364868)
Time Spent: 13h (was: 12h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 13h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364866
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 31/Dec/19 05:39
Start Date: 31/Dec/19 05:39
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10464:
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362142107
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
// (Required) The local name of this timer for the PTransform that
references it.
string local_name = 2;
}
+
+ // Settings that decide the coder type of wire coder.
+ message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
Review comment:
Please see the new PR:
https://github.com/apache/beam/pull/10484
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 364866)
Time Spent: 12h 50m (was: 12h 40m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364865 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 05:37 Start Date: 31/Dec/19 05:37 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-569856179 I can not reopen this PR, So , I have open a new PR here: https://github.com/apache/beam/pull/10484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364865) Time Spent: 12h 40m (was: 12.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 12h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364861 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 05:35 Start Date: 31/Dec/19 05:35 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464 In this PR would address multiple inputs/outputs for wire coder settings as suggested in https://github.com/apache/beam/pull/9979 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364850 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 03:47 Start Date: 31/Dec/19 03:47 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10484: [BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa… URL: https://github.com/apache/beam/pull/10484#issuecomment-569858254 I have rebase the code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364850) Time Spent: 12h 20m (was: 12h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364845
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 31/Dec/19 03:38
Start Date: 31/Dec/19 03:38
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10484:
[BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa…
URL: https://github.com/apache/beam/pull/10484#discussion_r362142926
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
// (Required) The local name of this timer for the PTransform that
references it.
string local_name = 2;
}
+
+ // Settings that decide the coder type of wire coder.
+ message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
Review comment:
Hi @lukecwik. Do you think we should add the WireCoderSetting property for
each input and output edge explicitly in the proto?
I was thinking the setting is not a necessary properties for each edge so
added the wire_coder_settings as an ExecutableStage level property. The
wire_coder_settings can be an empty list.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 364845)
Time Spent: 12h 10m (was: 12h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364844
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 31/Dec/19 03:37
Start Date: 31/Dec/19 03:37
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10484:
[BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa…
URL: https://github.com/apache/beam/pull/10484#discussion_r362142854
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
// (Required) The local name of this timer for the PTransform that
references it.
string local_name = 2;
}
+
+ // Settings that decide the coder type of wire coder.
+ message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 364844)
Time Spent: 12h (was: 11h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 12h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364841
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 31/Dec/19 03:28
Start Date: 31/Dec/19 03:28
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10464:
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362142107
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
// (Required) The local name of this timer for the PTransform that
references it.
string local_name = 2;
}
+
+ // Settings that decide the coder type of wire coder.
+ message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
Review comment:
Please see the new PR:
https://github.com/apache/beam/pull/10484
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 364841)
Time Spent: 11h 50m (was: 11h 40m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364839 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 03:25 Start Date: 31/Dec/19 03:25 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-569856179 I can not reopen this PR, So , I have open a new PR here: https://github.com/apache/beam/pull/10484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364839) Time Spent: 11h 40m (was: 11.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 11h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364837 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 03:06 Start Date: 31/Dec/19 03:06 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-569850965 Sorry, I accidentally deleted the branch, So, reopen the PR here: https://github.com/apache/beam/pull/10484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364837) Time Spent: 11.5h (was: 11h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364834 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 02:37 Start Date: 31/Dec/19 02:37 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-569850965 Sorry, I accidentally deleted the branch, So, reopen the PR here: https://github.com/apache/beam/pull/10484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364834) Time Spent: 11h 20m (was: 11h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 11h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364831 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 02:36 Start Date: 31/Dec/19 02:36 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10484: [BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa… URL: https://github.com/apache/beam/pull/10484 In this PR would address multiple inputs/outputs for wire coder settings as suggested in #9979 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364829 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 02:31 Start Date: 31/Dec/19 02:31 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364829) Time Spent: 11h (was: 10h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364805 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 31/Dec/19 00:01 Start Date: 31/Dec/19 00:01 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364805) Time Spent: 10h 50m (was: 10h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364757 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 30/Dec/19 21:05 Start Date: 30/Dec/19 21:05 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458#issuecomment-569796388 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364757) Time Spent: 10h 40m (was: 10.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364699 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 30/Dec/19 18:03 Start Date: 30/Dec/19 18:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458#issuecomment-569750414 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364699) Time Spent: 10.5h (was: 10h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 10.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364698
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 30/Dec/19 18:01
Start Date: 30/Dec/19 18:01
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10464: [BEAM-7951]
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362055043
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
// (Required) The local name of this timer for the PTransform that
references it.
string local_name = 2;
}
+
+ // Settings that decide the coder type of wire coder.
+ message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
Review comment:
We could add WireCoderSetting to the TimerId message explicitly instead of
having target here.
We could also add `input_wire_coder_setting` and make `outputs` a
`map`.
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 364698)
Time Spent: 10h 20m (was: 10h 10m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=363546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363546 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 26/Dec/19 11:46 Start Date: 26/Dec/19 11:46 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464#issuecomment-569044309 R: @lukecwik :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 363546) Time Spent: 10h 10m (was: 10h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=363528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363528 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 26/Dec/19 10:24 Start Date: 26/Dec/19 10:24 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10464: [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. URL: https://github.com/apache/beam/pull/10464 In this PR would address multiple inputs/outputs for wire coder settings as suggested in https://github.com/apache/beam/pull/9979 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=363395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363395 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 26/Dec/19 01:51 Start Date: 26/Dec/19 01:51 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458#issuecomment-568947230 R: @lukecwik :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 363395) Time Spent: 9h 50m (was: 9h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=363248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-363248 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 25/Dec/19 10:21 Start Date: 25/Dec/19 10:21 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458#issuecomment-568881078 Test failure due to [Beam-9026](https://issues.apache.org/jira/browse/BEAM-9026). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 363248) Time Spent: 9h 40m (was: 9.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362990 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 24/Dec/19 11:35 Start Date: 24/Dec/19 11:35 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458#issuecomment-568730726 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362990) Time Spent: 9.5h (was: 9h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362853 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 24/Dec/19 02:44 Start Date: 24/Dec/19 02:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10458: [BEAM-7951] Improve the docs for `beam_runner_api.proto` and `WindowedValue.java` URL: https://github.com/apache/beam/pull/10458 Fixes comments in beam_runner_api.proto and WindowedValue.java as suggested in [https://github.com/apache/beam/pull/9979#pullrequestreview-335982782](https://github.com/apache/beam/pull/9979#pullrequestreview-335982782) Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362852 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 24/Dec/19 02:37 Start Date: 24/Dec/19 02:37 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-568632164 Many thanks to the review and suggestions. @lukecwik @mxm Will open the follow-up PRs soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362852) Time Spent: 9h 10m (was: 9h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362691 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 23/Dec/19 19:56 Start Date: 23/Dec/19 19:56 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-568567369 I went through all the usages of FullWindowedValueCoder and didn't see any issues with any explicit downcasts being done where ParamWindowedvalueCoder wouldn't work in its place. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362691) Time Spent: 8h 50m (was: 8h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362692 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 23/Dec/19 19:56 Start Date: 23/Dec/19 19:56 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362692) Time Spent: 9h (was: 8h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362686
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360982221
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -637,7 +647,11 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
*
* A {@code ValueOnlyWindowedValueCoder} only encodes and decodes the
value. It drops timestamp
* and windows for encoding, and uses defaults timestamp, and windows for
decoding.
+ *
+ * @deprecated Use ParamWindowedValueCoder instead it is general purpose
implementation of the
Review comment:
```suggestion
* @deprecated Use ParamWindowedValueCoder instead, it is a general
purpose implementation of the
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362686)
Time Spent: 8h 20m (was: 8h 10m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362682
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360980308
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -674,6 +674,16 @@ message StandardCoders {
// Components: The element coder and the window coder, in that order
WINDOWED_VALUE = 8 [(beam_urn) = "beam:coder:windowed_value:v1"];
+// A windowed value coder with parameterized timestamp, windows and pane
info.
+// Encodes an element with only the value of the windowed value.
+// Decodes the value and assign the parameterized timestamp, windows and
PaneInfo to the
+// windowed value
Review comment:
```suggestion
// windowed value.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362682)
Time Spent: 8h (was: 7h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362689
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360982691
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +707,181 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp,
+ * windows, and pane info during encoding, and uses the supplied
parameterized timestamp, windows
+ * and pane info values during decoding when reconstructing the windowed
value.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
+
+private static final long serialVersionUID = 1L;
+
+private transient Instant timestamp;
+private transient Collection windows;
+private transient PaneInfo pane;
+
+private static final byte[] EMPTY_BYTES = new byte[0];
+
+/**
+ * Returns the {@link ParamWindowedValueCoder} from the given valueCoder,
windowCoder and the
+ * supplied parameterized timestamp, windows and pane info for {@link
WindowedValue}.
+ */
+public static ParamWindowedValueCoder of(
+Coder valueCoder,
+Coder windowCoder,
+Instant timestamp,
+Collection windows,
+PaneInfo pane) {
+ return new ParamWindowedValueCoder<>(valueCoder, windowCoder, timestamp,
windows, pane);
+}
+
+/**
+ * Returns the {@link ParamWindowedValueCoder} from the given valueCoder,
windowCoder and the
+ * supplied parameterized {@link BoundedWindow#TIMESTAMP_MIN_VALUE},
{@link #GLOBAL_WINDOWS} and
Review comment:
```suggestion
* {@link BoundedWindow#TIMESTAMP_MIN_VALUE} as the timestamp, {@link
#GLOBAL_WINDOWS} as the window, and
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362689)
Time Spent: 8.5h (was: 8h 20m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362683
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360980511
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -674,6 +674,16 @@ message StandardCoders {
// Components: The element coder and the window coder, in that order
WINDOWED_VALUE = 8 [(beam_urn) = "beam:coder:windowed_value:v1"];
+// A windowed value coder with parameterized timestamp, windows and pane
info.
+// Encodes an element with only the value of the windowed value.
+// Decodes the value and assign the parameterized timestamp, windows and
PaneInfo to the
+// windowed value
+// Components: The element coder and the window coder, in that order
+// The payload of this coder is an encoded windowed value using the
+// beam:coder:windowed_value:v1 coder parameterized by beam:coder:bytes:v1
Review comment:
```suggestion
// beam:coder:windowed_value:v1 coder parameterized by a
beam:coder:bytes:v1
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362683)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362688
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360982309
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +707,181 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
Review comment:
```suggestion
* A parameterized coder for {@code WindowedValue}.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362688)
Time Spent: 8h 20m (was: 8h 10m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362685=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362685
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360980545
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -674,6 +674,16 @@ message StandardCoders {
// Components: The element coder and the window coder, in that order
WINDOWED_VALUE = 8 [(beam_urn) = "beam:coder:windowed_value:v1"];
+// A windowed value coder with parameterized timestamp, windows and pane
info.
+// Encodes an element with only the value of the windowed value.
+// Decodes the value and assign the parameterized timestamp, windows and
PaneInfo to the
+// windowed value
+// Components: The element coder and the window coder, in that order
+// The payload of this coder is an encoded windowed value using the
+// beam:coder:windowed_value:v1 coder parameterized by beam:coder:bytes:v1
+// elements coder and the window coder that this param_windowed_value uses.
Review comment:
```suggestion
// element coder and the window coder that this param_windowed_value
coder uses.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362685)
Time Spent: 8h 10m (was: 8h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362687
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360982612
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +707,181 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp,
+ * windows, and pane info during encoding, and uses the supplied
parameterized timestamp, windows
+ * and pane info values during decoding when reconstructing the windowed
value.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
+
+private static final long serialVersionUID = 1L;
+
+private transient Instant timestamp;
+private transient Collection windows;
+private transient PaneInfo pane;
+
+private static final byte[] EMPTY_BYTES = new byte[0];
+
+/**
+ * Returns the {@link ParamWindowedValueCoder} from the given valueCoder,
windowCoder and the
+ * supplied parameterized timestamp, windows and pane info for {@link
WindowedValue}.
+ */
+public static ParamWindowedValueCoder of(
+Coder valueCoder,
+Coder windowCoder,
+Instant timestamp,
+Collection windows,
+PaneInfo pane) {
+ return new ParamWindowedValueCoder<>(valueCoder, windowCoder, timestamp,
windows, pane);
+}
+
+/**
+ * Returns the {@link ParamWindowedValueCoder} from the given valueCoder,
windowCoder and the
Review comment:
```suggestion
* Returns the {@link ParamWindowedValueCoder} for the given valueCoder
and windowCoder using
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362687)
Time Spent: 8h 20m (was: 8h 10m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362684
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360980265
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -674,6 +674,16 @@ message StandardCoders {
// Components: The element coder and the window coder, in that order
WINDOWED_VALUE = 8 [(beam_urn) = "beam:coder:windowed_value:v1"];
+// A windowed value coder with parameterized timestamp, windows and pane
info.
+// Encodes an element with only the value of the windowed value.
+// Decodes the value and assign the parameterized timestamp, windows and
PaneInfo to the
Review comment:
```suggestion
// Decodes the value and assigns the parameterized timestamp, windows
and pane info to the
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362684)
Time Spent: 8h 10m (was: 8h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362690
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 23/Dec/19 19:53
Start Date: 23/Dec/19 19:53
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r360982739
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +707,181 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp,
+ * windows, and pane info during encoding, and uses the supplied
parameterized timestamp, windows
+ * and pane info values during decoding when reconstructing the windowed
value.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
+
+private static final long serialVersionUID = 1L;
+
+private transient Instant timestamp;
+private transient Collection windows;
+private transient PaneInfo pane;
+
+private static final byte[] EMPTY_BYTES = new byte[0];
+
+/**
+ * Returns the {@link ParamWindowedValueCoder} from the given valueCoder,
windowCoder and the
+ * supplied parameterized timestamp, windows and pane info for {@link
WindowedValue}.
+ */
+public static ParamWindowedValueCoder of(
+Coder valueCoder,
+Coder windowCoder,
+Instant timestamp,
+Collection windows,
+PaneInfo pane) {
+ return new ParamWindowedValueCoder<>(valueCoder, windowCoder, timestamp,
windows, pane);
+}
+
+/**
+ * Returns the {@link ParamWindowedValueCoder} from the given valueCoder,
windowCoder and the
+ * supplied parameterized {@link BoundedWindow#TIMESTAMP_MIN_VALUE},
{@link #GLOBAL_WINDOWS} and
+ * {@link PaneInfo#NO_FIRING} for {@link WindowedValue}.
Review comment:
```suggestion
* {@link PaneInfo#NO_FIRING} as the pane info for parameters.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 362690)
Time Spent: 8h 40m (was: 8.5h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362667 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 23/Dec/19 19:14 Start Date: 23/Dec/19 19:14 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-568557618 Yeah, I'm back and wanted to go through this so please don't merge as I was looking for a large enough block of time to cover this change again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362667) Time Spent: 7h 50m (was: 7h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=362010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362010 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 21/Dec/19 01:45 Start Date: 21/Dec/19 01:45 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-568141245 Rebase code and squash the commits. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 362010) Time Spent: 7h 40m (was: 7.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=361649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361649 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 20/Dec/19 13:12 Start Date: 20/Dec/19 13:12 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-567919638 Yes, I have retried a few times but without success. I just find it is reported recently by [BEAM-8974](https://issues.apache.org/jira/browse/BEAM-8974) and it seems the PR will be merged soon. I have retried again. Hope we can make it this time by chance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 361649) Time Spent: 7.5h (was: 7h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=361647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361647 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 20/Dec/19 13:11 Start Date: 20/Dec/19 13:11 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-567919569 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 361647) Time Spent: 7h 20m (was: 7h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=361607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361607
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 20/Dec/19 11:47
Start Date: 20/Dec/19 11:47
Worklog Time Spent: 10m
Work Description: mxm commented on issue #9979: [BEAM-7951] Allow runner
to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#issuecomment-567898585
Seeing an error in the Python tests, but it looks unrelated:
```
09:41:25 ___ FnApiLogRecordHandlerTest.test_exc_info
09:41:25 [gw0] linux2 -- Python 2.7.12
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/target/.tox-py27-cython-pytest/py27-cython-pytest/bin/python
09:41:25
09:41:25 self =
09:41:25
09:41:25 def test_exc_info(self):
09:41:25 try:
09:41:25 raise ValueError('some message')
09:41:25 except ValueError:
09:41:25 _LOGGER.error('some error', exc_info=True)
09:41:25
09:41:25 self.fn_log_handler.close()
09:41:25
09:41:25 > log_entry =
self.test_logging_service.log_records_received[0].log_entries[0]
09:41:25 E IndexError: list index out of range
09:41:25
09:41:25 apache_beam/runners/worker/log_handler_test.py:110: IndexError
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361607)
Time Spent: 7h 10m (was: 7h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=360965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360965 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 17/Dec/19 16:13 Start Date: 17/Dec/19 16:13 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r358882174 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1136,6 +1136,62 @@ def __hash__(self): common_urns.coders.WINDOWED_VALUE.urn, WindowedValueCoder) +class ParamWindowedValueCoder(WindowedValueCoder): + """A coder used for parameterized Windowed values.""" + + def __init__(self, payload, components): +super(ParamWindowedValueCoder, self).__init__(components[0], components[1]) +self._wrapped_value_coder = components[0] +self.window_coder = components[1] +self.payload = payload +self.components = components + + def _create_impl(self): +return coder_impl.ParamWindowedValueCoderImpl( +self._wrapped_value_coder.get_impl(), +self.window_coder.get_impl(), +self.payload) + + def is_deterministic(self): +return all(c.is_deterministic() for c in [self._wrapped_value_coder, + self.window_coder]) + + def as_cloud_object(self, coders_context=None): Review comment: Seems fair to fix this at a later point in time, since this is not used in Dataflow at the moment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360965) Time Spent: 7h (was: 6h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=360964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360964
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 17/Dec/19 16:13
Start Date: 17/Dec/19 16:13
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9979: [BEAM-7951] Allow
runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r358880441
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -500,11 +505,22 @@ private void ensureWindowsAreASet() {
return FullWindowedValueCoder.of(valueCoder, windowCoder);
}
- /** Returns the {@code ValueOnlyCoder} from the given valueCoder. */
+ /**
+ * Returns the {@code ValueOnlyCoder} from the given valueCoder.
+ *
+ * @deprecated Use {@link #getParamWindowedValueCoder(Coder)} instead. The
{@link
+ * ValueOnlyWindowedValueCoder} is deprecated now.
+ */
+ @Deprecated
public static ValueOnlyWindowedValueCoder getValueOnlyCoder(Coder
valueCoder) {
Review comment:
I'm not sure we need to deprecate this method and replace all of the
invocations. This is still a useful method if no parameters are supplied. We
can simply redirect to the new ParamWindowedValueCoder here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360964)
Time Spent: 7h (was: 6h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=359430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359430 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 13/Dec/19 14:33 Start Date: 13/Dec/19 14:33 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-565462440 It seems that the CI fail not due to this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 359430) Time Spent: 6h 50m (was: 6h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=358416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358416 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 12/Dec/19 08:14 Start Date: 12/Dec/19 08:14 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-564898359 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358416) Time Spent: 6h 40m (was: 6.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=357031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357031 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 10/Dec/19 12:05 Start Date: 10/Dec/19 12:05 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-564003092 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357031) Time Spent: 6.5h (was: 6h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=356924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356924 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 10/Dec/19 09:28 Start Date: 10/Dec/19 09:28 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-563944280 @lukecwik @mxm Appreciate your review and nice comments. I have updated the PR according to your suggestions. I have shared more thoughts on some comments above. The other comments without reply are all make more sense to me :) Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356924) Time Spent: 6h 20m (was: 6h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=356922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356922 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 10/Dec/19 09:26 Start Date: 10/Dec/19 09:26 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-563944280 @lukecwik @mxm Appreciate your review and nice comments. I have updated the PR according to your suggestions. I have replied to some comments above. The other comments without reply are all make mor sense to me. Thank you very much! Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356922) Time Spent: 6h 10m (was: 6h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=356918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356918 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 10/Dec/19 09:19 Start Date: 10/Dec/19 09:19 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r355920792 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1136,6 +1136,62 @@ def __hash__(self): common_urns.coders.WINDOWED_VALUE.urn, WindowedValueCoder) +class ParamWindowedValueCoder(WindowedValueCoder): + """A coder used for parameterized Windowed values.""" + + def __init__(self, payload, components): +super(ParamWindowedValueCoder, self).__init__(components[0], components[1]) +self._wrapped_value_coder = components[0] +self.window_coder = components[1] +self.payload = payload +self.components = components + + def _create_impl(self): +return coder_impl.ParamWindowedValueCoderImpl( +self._wrapped_value_coder.get_impl(), +self.window_coder.get_impl(), +self.payload) + + def is_deterministic(self): +return all(c.is_deterministic() for c in [self._wrapped_value_coder, + self.window_coder]) + + def as_cloud_object(self, coders_context=None): Review comment: Thanks a lot for the detailed suggestions. How about implementing it in a following PR? We can throw an exception in this PR for now. I would not like to Involve too much in a single PR :). What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356918) Time Spent: 6h (was: 5h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=356916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356916
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 10/Dec/19 09:18
Start Date: 10/Dec/19 09:18
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #9979:
[BEAM-7951] Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r355920632
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +697,110 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp and
+ * windows for encoding, and uses defaults(constant) timestamp, and windows
for decoding.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
Review comment:
I was thinking `ParamWindowedValueCoder` is a special kind of
`FullWindowedValueCoder` with the parameterized timestamp, windows and pane
info for the WindowedValues, i.e., `ParamWindowedValueCoder` is an optimization
of `FullWindowedValueCoder`. We can also benefit from it such as reuse the code
of testing FullWindowedValueCoder in CommonCoderTest, etc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356916)
Time Spent: 5h 40m (was: 5.5h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=356917=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356917 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 10/Dec/19 09:18 Start Date: 10/Dec/19 09:18 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r355920792 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1136,6 +1136,62 @@ def __hash__(self): common_urns.coders.WINDOWED_VALUE.urn, WindowedValueCoder) +class ParamWindowedValueCoder(WindowedValueCoder): + """A coder used for parameterized Windowed values.""" + + def __init__(self, payload, components): +super(ParamWindowedValueCoder, self).__init__(components[0], components[1]) +self._wrapped_value_coder = components[0] +self.window_coder = components[1] +self.payload = payload +self.components = components + + def _create_impl(self): +return coder_impl.ParamWindowedValueCoderImpl( +self._wrapped_value_coder.get_impl(), +self.window_coder.get_impl(), +self.payload) + + def is_deterministic(self): +return all(c.is_deterministic() for c in [self._wrapped_value_coder, + self.window_coder]) + + def as_cloud_object(self, coders_context=None): Review comment: Thanks a lot for the detailed suggestions. How about implementing it in a following PR? We can throw an exception in this PR for now. Involve too much in a single PR seems not a good idea. What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356917) Time Spent: 5h 50m (was: 5h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=356915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356915
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 10/Dec/19 09:17
Start Date: 10/Dec/19 09:17
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #9979:
[BEAM-7951] Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r355920245
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1059,6 +1066,17 @@ message SideInput {
SdkFunctionSpec window_mapping_fn = 3;
}
+// Settings that decide the coder type of wire coder.
+message WireCoderSetting {
+ // (Required) The URN of the wire coder.
+ // Note that only windowed value coder or parameterized windowed value coder
are supported.
+ string urn = 1;
+
+ // (Optional) The data specifying any parameters to the URN. If
+ // the URN does not require any arguments, this may be omitted.
+ bytes payload = 2;
Review comment:
We can get the element coder id and window coder id like the same way of the
current windowed value coder. The element coder and window coder can be used to
encode and decode the payload. For example, in WireCoders, the elementCoderId
and
windowCoderId are get from PCollectionNode.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356915)
Time Spent: 5.5h (was: 5h 20m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354592 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Dec/19 19:05 Start Date: 05/Dec/19 19:05 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r354489900 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1136,6 +1136,62 @@ def __hash__(self): common_urns.coders.WINDOWED_VALUE.urn, WindowedValueCoder) +class ParamWindowedValueCoder(WindowedValueCoder): + """A coder used for parameterized Windowed values.""" Review comment: ```suggestion """A coder used for parameterized windowed values.""" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354592) Time Spent: 4h 20m (was: 4h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354603
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354476657
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -637,7 +637,11 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
*
* A {@code ValueOnlyWindowedValueCoder} only encodes and decodes the
value. It drops timestamp
* and windows for encoding, and uses defaults timestamp, and windows for
decoding.
+ *
+ * @deprecated Use ParamWindowedValueCoder instead as it is more
well-defined and can make
Review comment:
```suggestion
* @deprecated Use ParamWindowedValueCoder instead it is general purpose
implementation of the same concept but makes
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354603)
Time Spent: 5h 10m (was: 5h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354591
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354475918
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -500,9 +500,9 @@ private void ensureWindowsAreASet() {
return FullWindowedValueCoder.of(valueCoder, windowCoder);
}
- /** Returns the {@code ValueOnlyCoder} from the given valueCoder. */
- public static ValueOnlyWindowedValueCoder getValueOnlyCoder(Coder
valueCoder) {
-return ValueOnlyWindowedValueCoder.of(valueCoder);
+ /** Returns the {@code ParamWindowedValueCoder} from the given valueCoder. */
+ public static ParamWindowedValueCoder getValueOnlyCoder(Coder
valueCoder) {
Review comment:
This will break pipeline update for Dataflow since we will be changing the
type of coder being returned and used within existing users pipeline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354591)
Time Spent: 4h 10m (was: 4h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354606 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Dec/19 19:05 Start Date: 05/Dec/19 19:05 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r354489572 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -1088,6 +1088,54 @@ def get_estimated_size_and_observables(self, value, nested=False): return estimated_size, observables +class ParamWindowedValueCoderImpl(WindowedValueCoderImpl): + """For internal use only; no backwards-compatibility guarantees. + + A coder for windowed values with constant timestamp, windows and + pane info. The coder drops timestamp and windows for encoding, + and uses the constant timestamp, and windows for decoding.""" Review comment: ```suggestion and uses the constant timestamp, and windows during decoding.""" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354606) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354590
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354477598
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +697,110 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp and
+ * windows for encoding, and uses defaults(constant) timestamp, and windows
for decoding.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
+
+private final Instant timestamp;
+private final Collection windows;
+private final PaneInfo pane;
+
+public static ParamWindowedValueCoder of(
Review comment:
please add javadoc to the public static constructors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354590)
Time Spent: 4h 10m (was: 4h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354596
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354478177
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +697,110 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp and
+ * windows for encoding, and uses defaults(constant) timestamp, and windows
for decoding.
Review comment:
```suggestion
* windows, and pane info during encoding, and uses the supplied
parameterized timestamp, windows, and pane info values during decoding when
reconstructing the windowed value.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354596)
Time Spent: 4h 50m (was: 4h 40m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354594
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354476332
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -527,9 +527,9 @@ private void ensureWindowsAreASet() {
/** Coder for {@code WindowedValue}. */
public static class FullWindowedValueCoder extends WindowedValueCoder {
-private final Coder windowCoder;
+protected final Coder windowCoder;
Review comment:
why did you need to change the visibility of these two variables from
private to protected?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354594)
Time Spent: 4h 40m (was: 4.5h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354599=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354599
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354481565
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +697,110 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp and
+ * windows for encoding, and uses defaults(constant) timestamp, and windows
for decoding.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
Review comment:
Why did you want to extend FullWindowedValueCoder vs extending
WindowedValueCoder?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354599)
Time Spent: 5h (was: 4h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354595
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354471550
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1059,6 +1066,17 @@ message SideInput {
SdkFunctionSpec window_mapping_fn = 3;
}
+// Settings that decide the coder type of wire coder.
+message WireCoderSetting {
+ // (Required) The URN of the wire coder.
+ // Note that only windowed value coder or parameterized windowed value coder
are supported.
+ string urn = 1;
+
+ // (Optional) The data specifying any parameters to the URN. If
+ // the URN does not require any arguments, this may be omitted.
+ bytes payload = 2;
Review comment:
How are these parameters meant to be interpreted since you need to have the
encoded windowed value and also knowledge of the element coder id and window
coder id to construct a valid windowed_value and param_windowed_value coder?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354595)
Time Spent: 4h 40m (was: 4.5h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354597
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354474054
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
##
@@ -233,6 +246,20 @@ static ExecutableStage fromPayload(ExecutableStagePayload
payload) {
.map(id -> PipelineNode.pCollection(id,
components.getPcollectionsOrThrow(id)))
.collect(Collectors.toList());
return ImmutableExecutableStage.of(
-components, environment, input, sideInputs, userStates, timers,
transforms, outputs);
+components,
+environment,
+input,
+sideInputs,
+userStates,
+timers,
+transforms,
+outputs,
+wireCoderSetting);
}
+
+ // default wire coder, i.e., WINDOWED_VALUE coder
Review comment:
convert to javadoc comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354597)
Time Spent: 4h 50m (was: 4h 40m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354593=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354593
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354478249
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +697,110 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp and
Review comment:
```suggestion
* A {@code ParamWindowedValueCoder} only encodes and decodes the
value. It drops timestamp,
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354593)
Time Spent: 4.5h (was: 4h 20m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354605=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354605 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Dec/19 19:05 Start Date: 05/Dec/19 19:05 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r354490508 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1136,6 +1136,62 @@ def __hash__(self): common_urns.coders.WINDOWED_VALUE.urn, WindowedValueCoder) +class ParamWindowedValueCoder(WindowedValueCoder): + """A coder used for parameterized Windowed values.""" + + def __init__(self, payload, components): +super(ParamWindowedValueCoder, self).__init__(components[0], components[1]) +self._wrapped_value_coder = components[0] +self.window_coder = components[1] +self.payload = payload +self.components = components + + def _create_impl(self): +return coder_impl.ParamWindowedValueCoderImpl( +self._wrapped_value_coder.get_impl(), +self.window_coder.get_impl(), +self.payload) + + def is_deterministic(self): +return all(c.is_deterministic() for c in [self._wrapped_value_coder, Review comment: I believe you should only be checking the value coder similar to your java implementation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354605) Time Spent: 5h 20m (was: 5h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354600
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354476750
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -637,7 +637,11 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
*
* A {@code ValueOnlyWindowedValueCoder} only encodes and decodes the
value. It drops timestamp
* and windows for encoding, and uses defaults timestamp, and windows for
decoding.
+ *
+ * @deprecated Use ParamWindowedValueCoder instead as it is more
well-defined and can make
+ * timestamp, windows and PaneInfo configurable.
Review comment:
```suggestion
* timestamp, windows and pane info configurable.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354600)
Time Spent: 5h (was: 4h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354589
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354472665
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
##
@@ -124,6 +128,59 @@ public T fromComponents(List> components) {
};
}
+ static CoderTranslator>
paramWindowedValue() {
+return new CoderTranslator>() {
+ @Override
+ public List>
getComponents(WindowedValue.ParamWindowedValueCoder from) {
+return ImmutableList.of(from.getValueCoder(), from.getWindowCoder());
+ }
+
+ @Override
+ public byte[] getPayload(WindowedValue.ParamWindowedValueCoder from) {
+// Use FullWindowedValueCoder to encode the constant
members(timestamp, window, pane) in
+// ParamWindowedValueCoder
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+WindowedValue windowedValue =
+WindowedValue.of(new byte[] {}, from.getTimestamp(),
from.getWindows(), from.getPane());
Review comment:
nit: convert `new byte[] {}` to be a constant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354589)
Time Spent: 4h (was: 3h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354601
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354484482
##
File path:
model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
##
@@ -263,6 +263,20 @@ examples:
---
+# ParamWindowedValueCoder with constant value of:
+# timestamp: Instant.ofEpochMilli(1000)
+# windows: [IntervalWindow(10, 20)]
+# pane info: PaneInfo(false, true, PaneInfo.Timing.ON_TIME, 30, 40)
+coder:
+ urn: "beam:coder:param_windowed_value:v1"
+ payload:
"\x80\x00\x00\x00\x00\x00\x03è\x00\x00\x00\x01\x80\x00\x00\x00\x00\x00\x00\x14\n&\x1E(\x00"
+ components: [{urn: "beam:coder:varint:v1"},
+ {urn: "beam:coder:interval_window:v1"}]
+examples:
+ "\u0002": {value: 2}
Review comment:
Would it make sense to have the examples contain the timestamp, window and
pane info similar to the the windowed value above?
This would allow for multiple tests of the same urn with different payloads.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354601)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354604=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354604
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354473388
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
##
@@ -69,6 +72,14 @@
*/
Environment getEnvironment();
+ /**
+ * Returns the {@link WireCoderSetting} this stage executes in.
+ *
+ * An {@link WireCoderSetting} consists of settings which is used to
configure the type of the
Review comment:
```suggestion
* A {@link WireCoderSetting} consists of settings which is used to
configure the type of the
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354604)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354588
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354469091
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -672,6 +672,13 @@ message StandardCoders {
// Components: The element coder and the window coder, in that order
WINDOWED_VALUE = 8 [(beam_urn) = "beam:coder:windowed_value:v1"];
+// A windowed value coder with parameterized timestamp, windows and
PaneInfo.
Review comment:
I think we have to be explicit that the payload is an encoded windowed value
using the beam:coder:windowed_value:v1 coder parameterized by
beam:coder:bytes:v1 elements coder and the window coder that this
param_windowed_value uses.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354588)
Time Spent: 4h (was: 3h 50m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354598=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354598
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 05/Dec/19 19:05
Start Date: 05/Dec/19 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r354479963
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -693,4 +697,110 @@ public void registerByteSizeObserver(WindowedValue
value, ElementByteSizeObse
return Collections.singletonList(valueCoder);
}
}
+
+ /**
+ * A parameterized Coder for {@code WindowedValue}.
+ *
+ * A {@code ParamWindowedValueCoder} only encodes and decodes the value.
It drops timestamp and
+ * windows for encoding, and uses defaults(constant) timestamp, and windows
for decoding.
+ */
+ public static class ParamWindowedValueCoder extends
FullWindowedValueCoder {
+
+private final Instant timestamp;
Review comment:
mark these three as transient and instead override readObject/writeObject
methods so that you encode/decode the values using the coders.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354598)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354602 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Dec/19 19:05 Start Date: 05/Dec/19 19:05 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#discussion_r354492422 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1136,6 +1136,62 @@ def __hash__(self): common_urns.coders.WINDOWED_VALUE.urn, WindowedValueCoder) +class ParamWindowedValueCoder(WindowedValueCoder): + """A coder used for parameterized Windowed values.""" + + def __init__(self, payload, components): +super(ParamWindowedValueCoder, self).__init__(components[0], components[1]) +self._wrapped_value_coder = components[0] +self.window_coder = components[1] +self.payload = payload +self.components = components + + def _create_impl(self): +return coder_impl.ParamWindowedValueCoderImpl( +self._wrapped_value_coder.get_impl(), +self.window_coder.get_impl(), +self.payload) + + def is_deterministic(self): +return all(c.is_deterministic() for c in [self._wrapped_value_coder, + self.window_coder]) + + def as_cloud_object(self, coders_context=None): Review comment: This is meant to support Dataflow explicitly and is similar to the proto conversion. You'll need to add one to java in [CloudObjectTranslators](https://github.com/apache/beam/blob/a06b7a6cbad2386b7bee37d3be0d7f3f5b8bfaff/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java#L244) and wire it together. You'll need to add the encoded payload as part of the cloud object as well and I suggest you add it as a 'payload' field. Please use the same byte string -> json compatible string conversion found elsewhere in these as_cloud_object conversions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354602) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354060 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Dec/19 06:06 Start Date: 05/Dec/19 06:06 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-561985010 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354060) Time Spent: 3h 50m (was: 3h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=354006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354006 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Dec/19 02:16 Start Date: 05/Dec/19 02:16 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-561935444 Hi @lukecwik @mxm I have updated the PR. The changes as follows: The previous Proto definition of windowing information has been removed. Instead, I use the `Windowcoder` to encode the constant window information and store it in the payload of the `ParamWindowedValueCoder`. As for the coder name, I name it `ParamWindowedValueCoder` in the PR. We can further discuss about it if you still have other concerns. The last thing needs to be mentioned is it seems we have to make `BoundedWindow` and `PaneInfo` implement Serializable, because the coder(`ParamWindowedValueCoder`) needs to be serializable, So, I think the members should also be serializable. Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354006) Time Spent: 3h 40m (was: 3.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=352476=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352476 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 03/Dec/19 07:29 Start Date: 03/Dec/19 07:29 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-561037804 Hi @lukecwik, My first thoughts is that the `ValueOnly` seems give some hint that `timestamp/windows/pane` info are ignored. So, how about we rename it to `ParamWindowedValueCoder`(param_windowed_value_coder for the URN)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352476) Time Spent: 3.5h (was: 3h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=352105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352105 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 02/Dec/19 18:56 Start Date: 02/Dec/19 18:56 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-560531747 How about we stick with ValueOnlyWindowedValueCoder (value_only_windowed_value_coder for the URN), and repurpose the Java one to do this as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352105) Time Spent: 3h 20m (was: 3h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=351730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351730 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 02/Dec/19 04:26 Start Date: 02/Dec/19 04:26 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-560223830 Sorry for my belated reply as I have just returned from Flink Forward Asia 2019. (Talk about the PyFlink on Beam Portability Framework ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 351730) Time Spent: 3h 10m (was: 3h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=351729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351729 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 02/Dec/19 04:22 Start Date: 02/Dec/19 04:22 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-560223094 Hi, @mxm @lukecwik Thanks a lot for your review and suggestions. I think it's a good idea to use a `WindowedValue` coder to encode the constant values, and decode the `WindowedValue` at python side. In this way, we don't need to add protobuf class to define window info. I will try to update the PR according to your suggestions. As for the coder name, how about call it `constant_windowed_value`? I haven't found a better name yet. Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 351729) Time Spent: 3h (was: 2h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=349368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349368
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 25/Nov/19 21:25
Start Date: 25/Nov/19 21:25
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9979: [BEAM-7951]
Allow runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r350426810
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1338,3 +1359,44 @@ message ExecutableStagePayload {
string local_name = 2;
}
}
+
+// Window information in WindowedValue
+message WindowInfo {
+ // timestamp in milliseconds
+ int64 timestamp = 1;
+ repeated BoundedWindow bounded_windows = 2;
+ PaneInfo pane_info = 3;
+}
+
+// Represents window information assigned to data elements
+message BoundedWindow {
Review comment:
This limits the set of window types and the number of windows that can be
specified. I think we should go with a model where we encode a "dummy element"
which is replaced by the actual element when decoding. For example, the payload
would be:
```
message ParameterizedWindowedValuePayload {
// A specification of a windowed value coder where the element coder is
always "beam:coder:bytes:v1".
string coder_id;
// Contains an encoded windowed value with an empty byte[] element. This
elements windowing
// information should be copied to all elements that are decoded with this
coder.
bytes value;
}
```
This allows us to use the full range of window encodings and the already
defined pane/timestamp encodings.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 349368)
Time Spent: 2h 50m (was: 2h 40m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=349196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349196
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 25/Nov/19 15:50
Start Date: 25/Nov/19 15:50
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9979: [BEAM-7951] Allow
runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r350263969
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/BoundedWindow.java
##
@@ -35,7 +36,7 @@
* windows that are logically equal will be treated as equal by {@code
equals()} and {@code
* hashCode()}.
*/
-public abstract class BoundedWindow {
+public abstract class BoundedWindow implements Serializable {
Review comment:
Not sure about this one. Why can't we use `WindowCoder` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 349196)
Time Spent: 2.5h (was: 2h 20m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=349197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349197
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 25/Nov/19 15:50
Start Date: 25/Nov/19 15:50
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9979: [BEAM-7951] Allow
runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r350264092
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
##
@@ -40,7 +41,7 @@
*
* Note: This does not uniquely identify a pane, and should not be used for
comparisons.
*/
-public final class PaneInfo {
+public final class PaneInfo implements Serializable {
Review comment:
Not sure about this one. Why can't we use `PaneInfoCoder` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 349197)
Time Spent: 2h 40m (was: 2.5h)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=349195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349195
]
ASF GitHub Bot logged work on BEAM-7951:
Author: ASF GitHub Bot
Created on: 25/Nov/19 15:50
Start Date: 25/Nov/19 15:50
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9979: [BEAM-7951] Allow
runner to configure customization WindowedValue coder.
URL: https://github.com/apache/beam/pull/9979#discussion_r350257885
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1338,3 +1359,44 @@ message ExecutableStagePayload {
string local_name = 2;
}
}
+
+// Window information in WindowedValue
+message WindowInfo {
+ // timestamp in milliseconds
+ int64 timestamp = 1;
+ repeated BoundedWindow bounded_windows = 2;
+ PaneInfo pane_info = 3;
+}
+
+// Represents window information assigned to data elements
+message BoundedWindow {
+ oneof window {
+GlobalWindow global_window = 1;
+IntervalWindow interval_window = 2;
Review comment:
What about session windows?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 349195)
Time Spent: 2h 20m (was: 2h 10m)
> Allow runner to configure customization WindowedValue coder such as
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and
> pane properties in Flink and so it will be better to make the coder
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=345774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345774 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 19/Nov/19 02:43 Start Date: 19/Nov/19 02:43 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-555306699 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345774) Time Spent: 2h 10m (was: 2h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=345079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345079 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 18/Nov/19 06:25 Start Date: 18/Nov/19 06:25 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-554871240 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345079) Time Spent: 2h (was: 1h 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=345056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345056 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 18/Nov/19 03:25 Start Date: 18/Nov/19 03:25 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-554834870 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345056) Time Spent: 1h 50m (was: 1h 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=345048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345048 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 18/Nov/19 02:08 Start Date: 18/Nov/19 02:08 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-554820488 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345048) Time Spent: 1h 40m (was: 1.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=345040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345040 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 18/Nov/19 01:41 Start Date: 18/Nov/19 01:41 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-554815776 Hi @mxm, Thanks a lot for your attention! I have updated the PR! :) The changes mainly include: 1. Add a ParameterizedWindowedValueCoder that with constant timestamp, windows and paneInfo. The ParameterizedWindowedValueCoder at the harness side init the constants from the payload of the coder, and to achieve this, the `message WindowInfo` is added in `beam_runner_api.proto` to carry the constant information. 2. Add WireCoderSetting in ExecutableStagePayload. With the WireCoderSetting, we can configure wire coder to `ParameterizedWindowedValueCoder` to avoid redundant encoding and decoding. Would be great if you can take a look! Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345040) Time Spent: 1.5h (was: 1h 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=345039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345039 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 18/Nov/19 01:41 Start Date: 18/Nov/19 01:41 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-554815776 Hi @mxm, Thanks a lot for your attention! I have updated the PR. The changes mainly include: 1. Add a ParameterizedWindowedValueCoder that with constant timestamp, windows and paneInfo. The ParameterizedWindowedValueCoder at the harness side init the constants from the payload of the coder, and to achieve this, the `message WindowInfo` is added in `beam_runner_api.proto` to carry the constant information. 2. Add WireCoderSetting in ExecutableStagePayload. With the WireCoderSetting, we can configure wire coder to `ParameterizedWindowedValueCoder` to avoid redundant encoding and decoding. Would be great if you can take a look! Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345039) Time Spent: 1h 20m (was: 1h 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=343289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-343289 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 14/Nov/19 10:05 Start Date: 14/Nov/19 10:05 Worklog Time Spent: 10m Work Description: mxm commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-553816784 Please let me know once you have updated the PR according to the discussion result. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 343289) Time Spent: 1h 10m (was: 1h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=339252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-339252 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 06/Nov/19 09:13 Start Date: 06/Nov/19 09:13 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-550217664 The discussion mail thread: https://lists.apache.org/thread.html/a93e3985cd729b3cb5a8bb9fbff59851d075d5894b411e2ffaedf999@%3Cdev.beam.apache.org%3E This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 339252) Time Spent: 1h (was: 50m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=339230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-339230 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 06/Nov/19 08:57 Start Date: 06/Nov/19 08:57 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-550211327 Hi, @lukecwik Thanks a lot for your advice. I think a discussion in mailing list is a good idea. I will fire a discussion later. As for the second option listed by you, the PR is quite similar to it. With the PR, the (Python)SDK realize it was given a value only coder from the coder_id in `RemoteGrpcPort`. And it seems we don't need to change `BeamFnDataReadRunner` or `BeamFnDataWriteRunner` due to `ValueOnlyWindowedValueCoder` is also a coder of WindowedValue. What do you think? Best, Jincheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 339230) Time Spent: 50m (was: 40m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=338574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338574 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Nov/19 07:24 Start Date: 05/Nov/19 07:24 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-549696672 R: @lukecwik @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338574) Time Spent: 40m (was: 0.5h) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=338466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338466 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 05/Nov/19 02:53 Start Date: 05/Nov/19 02:53 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-549334516 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338466) Time Spent: 0.5h (was: 20m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=338088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338088 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 04/Nov/19 12:36 Start Date: 04/Nov/19 12:36 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979#issuecomment-549334516 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338088) Time Spent: 20m (was: 10m) > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=338041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338041 ] ASF GitHub Bot logged work on BEAM-7951: Author: ASF GitHub Bot Created on: 04/Nov/19 09:20 Start Date: 04/Nov/19 09:20 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #9979: [BEAM-7951] Allow runner to configure customization WindowedValue coder. URL: https://github.com/apache/beam/pull/9979 Motivation: The coder of WindowedValue cannot be configured and it’s always FullWindowedValueCoder. We don't need to serialize the timestamp, window and pane properties in Flink and so it will be better to make the coder configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) Detail changes: - Add ValueOnlyWindowedValueCoder into MODEL_CODER so that we can use value only coder between runner and harness. - To make it configurable and also maybe for other configurations in futher(i.e., more general), we add a `configurations` map in Components. - Add python ValueOnlyWindowedValueCoder. Some concerns reviewers may have: - Do we have any other options instead of adding the configurations in Components(beam_runner_api.proto). Yes, we have other options. For example, we can add some flags in the methods when call `addSdkWireCoder` and `addRunnerWireCoder`. However, the call stack is very deep(from DefaultJobBundleFactory.forStage() to WireCoders.addSdkWireCoder()). It would be a mess if we add a flag for all these methods. - The current changes are not used in Beam project. Due to the doFn can emit timestamp in beam, it seems we can't tell when we can ignore encode or decode the timestamp field, i.e., currently, we only use FullWindowedValueCoder in Beam. However, in Flink we don't have such problem and can make use of the ValueOnlyWindowedValueCoder. - Have not add support for other SDKs(go)? Maybe we can add support for other SDKs later when we find it is useful? Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[