[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100703#comment-16100703
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3580


> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093728#comment-16093728
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3586


> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091938#comment-16091938
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3587


> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091856#comment-16091856
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3587

Change PR template from 1234 to XXX

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

@jkff PTAL


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam Fix-PR-Template

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3587


commit b827f65622f9cd9203803b76935aac422c179803
Author: Sourabh Bajaj 
Date:   2017-07-18T17:30:44Z

Change PR template from 1234 to XXX




> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091822#comment-16091822
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3586

Increase the gRPC message size to max value

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
The default message size that can be sent or received over gRPC is 4MB. For 
the fn API case we do buffering at a layer above, so setting the message size 
at the gRPC level to a max value. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam msg_size

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3586


commit 2814ed32b79c53451635b8cbc5b264b2f6c0837c
Author: Vikas Kedigehalli 
Date:   2017-07-18T17:06:46Z

Increase the gRPC message size to max value




> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090765#comment-16090765
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3580

Let IsBounded take True value.

This is useful for languages like Python that may use this in a conditional 
statement (or allow assignment from True->1/False->0).

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam patch-9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3580.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3580


commit 0f0d2df88db40ad2b5b3893bf152b6221d68ca50
Author: Robert Bradshaw 
Date:   2017-07-17T23:01:18Z

Let IsBounded take True value.

This is useful for languages like Python that may use this in a conditional 
statement (or allow assignment from True->1/False->0).




> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15790204#comment-15790204
 ] 

Davor Bonaci commented on BEAM-1234:


There are other alternatives too, i.e., checkpoint() -- or a hint vs. a 
requirement. At this point, I tend to prefer a required checkpoint, but a 
detailed analysis would be useful.

> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)