[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance
[
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576406#comment-15576406
]
Jeffrey Payne commented on BEAM-697:
Submitted fix in PR https://github.com/apache/incubator-beam/pull/
> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}}
> both pass a literal value of {{false}} to the constructor's {{validate}}
> parameter for the returned {{Bound}} instance. This could, obviously, lead
> to unintended behavior wrt pipeline validation. Looks like a simple copy and
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included
> the *trivial* fix for this bug it in that set of changes. Creating this in
> case my PR is not accepted right away. Please close if this issue is wholly
> unnecessary. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance
[
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575414#comment-15575414
]
Jeffrey Payne commented on BEAM-697:
Will do!
> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}}
> both pass a literal value of {{false}} to the constructor's {{validate}}
> parameter for the returned {{Bound}} instance. This could, obviously, lead
> to unintended behavior wrt pipeline validation. Looks like a simple copy and
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included
> the *trivial* fix for this bug it in that set of changes. Creating this in
> case my PR is not accepted right away. Please close if this issue is wholly
> unnecessary. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance
[
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539503#comment-15539503
]
Jeffrey Payne commented on BEAM-697:
The fix for this is included in
https://github.com/apache/incubator-beam/pull/1027, pushed a few minute ago,
FYI. I added two unit tests to cover the two specific use cases.
> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}}
> both pass a literal value of {{false}} to the constructor's {{validate}}
> parameter for the returned {{Bound}} instance. This could, obviously, lead
> to unintended behavior wrt pipeline validation. Looks like a simple copy and
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included
> the *trivial* fix for this bug it in that set of changes. Creating this in
> case my PR is not accepted right away. Please close if this issue is wholly
> unnecessary. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance
[
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15538889#comment-15538889
]
Jeffrey Payne commented on BEAM-697:
[~frances], let me know if you want me to *not* include the trivial fix for
this in my forthcoming changes for BEAM-55 in
https://github.com/apache/incubator-beam/pull/1027. Thanks.
> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}}
> both pass a literal value of {{false}} to the constructor's {{validate}}
> parameter for the returned {{Bound}} instance. This could, obviously, lead
> to unintended behavior wrt pipeline validation. Looks like a simple copy and
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included
> the *trivial* fix for this bug it in that set of changes. Creating this in
> case my PR is not accepted right away. Please close if this issue is wholly
> unnecessary. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance
Jeffrey Payne created BEAM-697:
--
Summary: TextIO.Write.Bound.withHeader() and withFooter() do not
use the current value of validate for the returned Bound instance
Key: BEAM-697
URL: https://issues.apache.org/jira/browse/BEAM-697
Project: Beam
Issue Type: Bug
Components: beam-model
Affects Versions: 0.2.0-incubating, 0.3.0-incubating
Reporter: Jeffrey Payne
Assignee: Frances Perry
Priority: Minor
Fix For: 0.3.0-incubating
{{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}}
both pass a literal value of {{false}} to the constructor's {{validate}}
parameter for the returned {{Bound}} instance. This could, obviously, lead to
unintended behavior wrt pipeline validation. Looks like a simple copy and
paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
I'm about to push another iteration for a PR for BEAM-55 and have included the
*trivial* fix for this bug it in that set of changes. Creating this in case my
PR is not accepted right away. Please close if this issue is wholly
unnecessary. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-55) Allow users to compress FileBasedSink output files
[ https://issues.apache.org/jira/browse/BEAM-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531516#comment-15531516 ] Jeffrey Payne commented on BEAM-55: --- We too prefer to use binary file formats like Avro or Parquet, for many reasons, including automatic compression handling. Unfortunately, we have several existing SLAs with clients that necessitate compressed CSV output, some even require a *single compressed CSV file*, ugh. What they do with the file once it's out of our hands is their problem :) I'll read through the contribution guide, fork beam, and submit a PR. Thanks again for the direction! > Allow users to compress FileBasedSink output files > -- > > Key: BEAM-55 > URL: https://issues.apache.org/jira/browse/BEAM-55 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Daniel Halperin >Priority: Minor > > FileBasedSink (also TextIO.Write, AvroIO.Write, etc). does not have an option > for compressing its output. > In general, we discourage compression because it limits or blocks scalably > reading from a file in parallel. However, users may want it -- so we should > support the option (with appropriate warnings). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
