[jira] [Commented] (BEAM-658) Support Read.Unbounded primitive

2016-10-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609213#comment-15609213
 ] 

ASF GitHub Bot commented on BEAM-658:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1143


> Support Read.Unbounded primitive
> 
>
> Key: BEAM-658
> URL: https://issues.apache.org/jira/browse/BEAM-658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Spark runner support for Read.Unbounded primitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-658) Support Read.Unbounded primitive

2016-10-20 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591725#comment-15591725
 ] 

ASF GitHub Bot commented on BEAM-658:
-

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1143

[BEAM-658] Support Read.Unbounded primitive.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Changed mapSourceFunction to use scala's native Option.

Upgrade to kryo-serializers 0.39 that provides support for ReverseList 
(used by Top).

Better logging.

Allow a longer read time-frame for read in tests.

Assert initial parallelism is gerater than zero.

Add OnBatchCompleted listener that writes to Kafka.

Test latest.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-658

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1143


commit 71ffb6b6aca9c997326d3b15951d1f295aaf7505
Author: Sela 
Date:   2016-09-29T13:30:32Z

[BEAM-658] Support Read.Unbounded primitive.

Changed mapSourceFunction to use scala's native Option.

Upgrade to kryo-serializers 0.39 that provides support for ReverseList 
(used by Top).

Better logging.

Allow a longer read time-frame for read in tests.

Assert initial parallelism is gerater than zero.

Add OnBatchCompleted listener that writes to Kafka.

Test latest.




> Support Read.Unbounded primitive
> 
>
> Key: BEAM-658
> URL: https://issues.apache.org/jira/browse/BEAM-658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Spark runner support for Read.Unbounded primitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-658) Support Read.Unbounded primitive

2016-09-29 Thread Amit Sela (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534212#comment-15534212
 ] 

Amit Sela commented on BEAM-658:


WIP branch is available here: 
https://github.com/amitsela/incubator-beam/tree/BEAM-658-WIP

> Support Read.Unbounded primitive
> 
>
> Key: BEAM-658
> URL: https://issues.apache.org/jira/browse/BEAM-658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Spark runner support for Read.Unbounded primitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-658) Support Read.Unbounded primitive

2016-09-21 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/BEAM-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510215#comment-15510215
 ] 

Jean-Baptiste Onofré commented on BEAM-658:
---

+1

IMHO, it would be great to implement it on master to be able to include it in 
the next release (on Spark 1.x runner).

> Support Read.Unbounded primitive
> 
>
> Key: BEAM-658
> URL: https://issues.apache.org/jira/browse/BEAM-658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Spark runner support for Read.Unbounded primitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-658) Support Read.Unbounded primitive

2016-09-21 Thread Amit Sela (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509674#comment-15509674
 ] 

Amit Sela commented on BEAM-658:


Suggested design: 
https://docs.google.com/document/d/12BzHbETDt7ICIF7vc8zzCeLllmIpvvaVDIdBlcIwE1M/edit?usp=sharing.

> Support Read.Unbounded primitive
> 
>
> Key: BEAM-658
> URL: https://issues.apache.org/jira/browse/BEAM-658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Spark runner support for Read.Unbounded primitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)