[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768525#comment-15768525
 ] 

ASF GitHub Bot commented on BEAM-469:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1680

[BEAM-XXX] Make KVCoder more efficient by removing unnecessary nesting

See [BEAM-469] for more information about why this is
correct.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
efficient-nested-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1680.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1680


commit 621e8250c9535d773c4f4440a34ea0833912b51f
Author: Dan Halperin 
Date:   2016-12-21T23:37:49Z

[BEAM-XXX] Make KVCoder more efficient by removing unnecessary nesting

See [BEAM-469] for more information about why this is
correct.




> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: 0.3.0-incubating
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768476#comment-15768476
 ] 

Daniel Halperin commented on BEAM-469:
--

Sorry I missed this JIRA comment, [~mariusz89016]! A bit late, but...

Say a coder C does not have the nested context. Then we actually have the 
guarantee that no one will put later elements.

So if {{NullableCoder}} does not have the nested context, then no one will put 
more elements after whatever the {{NullableCoder}} puts. If the NC puts {{0}} 
then that's it -- the element is null. But if the NC puts {{1}}, then we know 
that all remaining bytes in the encoded string belong to the inner coder. That 
is effectively saying that the inner coder also does not need to have the 
nested context, so it does not need to write its own length.

In your example, the {{NullableCoder}} is used in an inner context. So the 
inner coder needs to also use the inner context, because there may be more 
encoded elements later.

In either case: the context of the nullable coder can be the same as the 
context of the inner coder. This is why in the patch here, we simply pass the 
NC's context down into the inner coder. All we have removed is the _additional_ 
nesting that was used.

https://patch-diff.githubusercontent.com/raw/apache/incubator-beam/pull/992.patch
 

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: 0.3.0-incubating
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583205#comment-15583205
 ] 

Daniel Halperin commented on BEAM-469:
--

Backwards compatibility: this is a change to the encoding of data, which will 
only affect Cloud Dataflow streaming pipelines across update. This is not yet 
part of the Beam model, and no one in Beam depends on it.

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: 0.3.0-incubating
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)