[
https://issues.apache.org/jira/browse/BEAM-11457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252659#comment-17252659
]
Jozef Vilcek commented on BEAM-11457:
-------------------------------------
It is basically altering a behavior of `BoundedSource` of the IO, therefore
builder API seem more fitting to me than hadoop configuration, but since this
is internal optimisation I have no problem to move it there if you prefer.
As for separate clone for key / value, I decided for one flag only because of
my use case and thought it is better added when it is needed. But it make sense
to add it for consistency if you prefer that to have from the beginning.
Making immutable types list configurable and smarter felt more complicated than
turning clone on/off. Usually user is well aware of what kind of data he uses
and it's properties. I am using transforms where I already step into the Beam
model and do not use mutable elements as it should be with Beam. But making
such change breaks backward compatibility.
> Enable skip key-value clone for HadoopFormatIO
> -----------------------------------------------
>
> Key: BEAM-11457
> URL: https://issues.apache.org/jira/browse/BEAM-11457
> Project: Beam
> Issue Type: Improvement
> Components: io-java-hadoop-format
> Affects Versions: 2.25.0
> Reporter: Jozef Vilcek
> Assignee: Jozef Vilcek
> Priority: P3
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HadoopFormatIO eagerly clone key-values if they are not a well known
> immutable types. This make sense due to how hadoop Writables behave. However,
> user can use key value translation functions which possibly already output
> immutable types. In such case it would be of benefit if extra clone via coder
> can be avoided.
> It would be great if coder can be consulted on the type an it's need for
> clone. However I am not aware if such detection is possible. I propose to add
> config parameter for skipping the clone which can be used by IO user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
