[GitHub] incubator-beam pull request #1444: [BEAM-1057] Fix JDBC test derby startup i...

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1444

[BEAM-1057] Fix JDBC test derby startup issues

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam BEAM-1057

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1444


commit 935553f88f327598320c4f5c1088d490a8a7e8e2
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-11-28T21:47:37Z

[BEAM-1057] Fix JDBC test derby startup issues




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1415: [BEAM-1034] Clean up tmp area in tests

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1415

[BEAM-1034] Clean up tmp area in tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam sorter-tmpleak

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1415






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1324: [BEAM-948] Add ability to write files to ...

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1324

[BEAM-948] Add ability to write files to GridFS

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam gridfs-sink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1324.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1324


commit 9f10282e04e5fcaace576217e725cb73022d8994
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-11-07T22:21:55Z

[BEAM-948] Add ability to write files to GridFS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1265: [BEAM-822] Move the separate package-info...

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1265

[BEAM-822] Move the separate package-info.java compile to java7 profile

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Move the separate package-info.java compile to java7 profile since it's not 
needed with java8
This fixes compiling in Eclipse (assuming Neon which require java8)
Slightly speeds up Java8 compile (one invoke of compiler plugin)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam eclipse-packageinfo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1265.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1265


commit f6efec86fa6301c166a25316490e65900e46fc3c
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-11-02T20:14:00Z

Move the separate package-info.java compile to java7 profile since it's not 
needed with java8
This fixes compiling in Eclipse (assuming Neon which require java8)
Slightly speeds up Java8 compile (one invoke of compiler plugin)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1094: Build in eclipse/eclipse-jdt

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1094

Build in eclipse/eclipse-jdt

This just updates the code to build with eclipse-jdt (again) and adds the 
travis.xml changes back in .

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam eclipse2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1094


commit a8cbd246c5f74e2c6066f424db19d84ad316b06e
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-10-13T16:41:32Z

Update to compile in eclipse again

commit 83f0b6436fb5a800f7ef0794aa65eb4d5a029069
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-14T17:21:34Z

Attempt to add eclipse-jdt profile to a travis build




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[incubator-beam] Git Push Summary

[dkulp]

Repository: incubator-beam
Updated Branches:
  refs/heads/eclipse2 [deleted] 83f0b6436

[2/2] incubator-beam git commit: Attempt to add eclipse-jdt profile to a travis build

[dkulp]

Attempt to add eclipse-jdt profile to a travis build


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/83f0b643
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/83f0b643
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/83f0b643

Branch: refs/heads/eclipse2
Commit: 83f0b6436fb5a800f7ef0794aa65eb4d5a029069
Parents: a8cbd24
Author: Daniel Kulp 
Authored: Wed Sep 14 13:21:34 2016 -0400
Committer: Daniel Kulp 
Committed: Thu Oct 13 12:41:44 2016 -0400

--
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/83f0b643/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 786b370..5133a43 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -39,6 +39,8 @@ matrix:
 - os: linux
   env: CUSTOM_JDK="oraclejdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 - os: linux
+  env: CUSTOM_JDK="oraclejdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m' -Peclipse-jdt"
+- os: linux
   env: CUSTOM_JDK="openjdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 
 before_install:

[1/2] incubator-beam git commit: Update to compile in eclipse again

[dkulp]

Repository: incubator-beam
Updated Branches:
  refs/heads/eclipse2 [created] 83f0b6436


Update to compile in eclipse again


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a8cbd246
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a8cbd246
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a8cbd246

Branch: refs/heads/eclipse2
Commit: a8cbd246c5f74e2c6066f424db19d84ad316b06e
Parents: 7322616
Author: Daniel Kulp 
Authored: Thu Oct 13 12:41:32 2016 -0400
Committer: Daniel Kulp 
Committed: Thu Oct 13 12:41:32 2016 -0400

--
 .../beam/runners/direct/BoundedReadEvaluatorFactory.java   | 3 ++-
 .../apache/beam/runners/direct/TestStreamEvaluatorFactory.java | 2 +-
 .../beam/runners/direct/UnboundedReadEvaluatorFactory.java | 5 +++--
 .../main/java/org/apache/beam/runners/spark/io/SourceRDD.java  | 6 --
 4 files changed, 10 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8cbd246/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 326a535..8e06c3f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -33,6 +33,7 @@ import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 
 /**
@@ -131,7 +132,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 }
 
 private  Collection createInitialSplits(
-AppliedPTransform> transform) {
+AppliedPTransform transform) {
   BoundedSource source = transform.getTransform().getSource();
   return Collections.singleton(
   evaluationContext

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8cbd246/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
index 4a48a58..fc4f7c5 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
@@ -204,7 +204,7 @@ class TestStreamEvaluatorFactory implements 
TransformEvaluatorFactory {
 }
 
 private  Collection createInputBundle(
-AppliedPTransform> transform) {
+AppliedPTransform transform) {
   CommittedBundle initialBundle =
   evaluationContext
   .createRootBundle()

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8cbd246/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index 08dc286..183225f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -37,6 +37,7 @@ import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.joda.time.Instant;
 
@@ -70,7 +71,7 @@ class UnboundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
   }
 
   private  TransformEvaluator createEvaluator(
-  AppliedPTransform, Read.Unbounded> 
application) {
+  

[GitHub] incubator-beam pull request #1025: [BEAM-674] Gridfs Source refactoring

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1025

[BEAM-674] Gridfs Source refactoring

Refactor of the GridFS based Source based on feedback from @jkff 

BoundedSource is now a source of ObjectID's and a separate DoFn is used to 
convert/parse the GridFSDBFile into usable chunks.   

Testcase for splitting added.

Variables not needed by the Source are pulled out and stuck on the 
transform instead.

Optimized the non-split case a bit by not querying all the ObjectIds up 
front.  

Optimize unit tests by setting up test data per class instead of per test.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam gridfs-t2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1025


commit 5aad971bcd1d32ba06cec9d4870e7aa9e9dc17f5
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T02:44:37Z

Split BoundedSource into a BoundedSource and a DoFn<...>

commit 2fc219cdd33e89d65d457dd3767bd378ffc0
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T13:03:31Z

Optimize reading for non-split case

commit e58fc61868988cc40c325d913fca37b26e3db99c
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T13:18:17Z

Use objectId timestamp

commit ed73d77b21651d6ef1d8cf2892dc267794d52d10
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T13:57:44Z

Pull parser out of BoundedSource, add maxSkew

commit 277667527cf0a23704b3ae3d05b2c8e2c2bcea3c
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T14:48:42Z

Add test case for the split

commit db30aabac4629ae167e4ede73de79257b4a93336
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T15:00:44Z

Don't need the generic on the Source and Reader

commit 1cdb2ce716b7e020c5306494b414b5bb136abb24
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-29T16:29:51Z

Rename maxSkew to allowedTimestampSkew to match other DoFn's




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1003: [BEAM-674] Source part of GridFS IO

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1003

[BEAM-674] Source part of GridFS IO

This is the "Source" part for GridFS based IO for beam.  (will work on Sink 
next, but would like to get this reviewed and merged first) . The "default" is 
to parse each file as text files (by line), but a parser function can be 
provided to take the InputStream and parse via whatever is required.   

For runners that can split into bundles it attempts to assign files in the 
grid to different bundles.  


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam gridfs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1003


commit d5cdc2429622f65a762774de8b5baf15334e55e2
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-16T20:58:56Z

Add GridFS io

commit a9212662744c14f10cd811540c3e9268c32c25c4
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-16T21:19:50Z

Fix checkstyle issues

commit cee0a06b6a465a276c2c5410d7d3f9af703982d4
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T17:22:44Z

Attempt to get a converter in there

commit fafa8fa607f22eacf918abb13419f28df9d2a8e9
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T17:32:44Z

Fix javac compile problem

commit 7e9872f12c74902f1a23e5a27eb0027ae753947a
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T17:50:11Z

Force a serializable

commit 265747946864b226235ee5b758e6c10b7cc3992f
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T17:56:03Z

Add the needed coder

commit 4f54495afe7ff4768d873350c345d39905d812fc
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T18:02:39Z

Change to using the GridFSDBFile instead of InputStream so the parsingFn 
can have access to tall the metadata

commit cbeebf02542a5e5a5f4b9a6c370b1b68b46d2deb
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T18:25:23Z

Flip to allowing the parser to have complete control over how the item is 
added to the collection

commit a08007b9f444fedcde78ab38c6cdf505b3864c61
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-19T18:26:19Z

Fix unused imports

commit a4840e98d891d3fa783654a472af06c4d399a929
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-21T19:51:45Z

Add test for the parser functionality and cleanup some of that code

commit 438a792a796be77186d79aa3fdb221efcced6d4f
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-21T20:01:33Z

Move the coder out from the parser

commit e8fcdbf3cebd6fa4648f328484dee07fec35b21a
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-22T12:36:50Z

Fix test

commit 1d1a373fc7cec4e78bf0e618a902c15005fc36b4
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-23T14:49:53Z

Flip to using BoundedSource so it can be broken up into bundles




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #949: Update code to import into eclipse using m...

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/949

Update code to import into eclipse using m2e and compile (checkstyle …

I'm trying to get beam to load into Eclipse (Neon) using m2e.   Currently, 
there are a bunch of compile errors which prevents it from building within 
Eclipse due to some differences between the standard Oracle "javac" compiler 
and Eclipse's ejc compiler with certain generics usages.   Also, the eclipse 
compiler complains if there are two package-info.java files in the same package 
(usually due to one in the "test" tree that duplicates the package from the 
non-test tree) so I eliminated the ones in the test tree.  

This is a start.  With this, beam will import into Eclipse, but then 
generates a TON of Checkstyle errors due to checkstyle checking the avro 
generated code.   You can right click on the projects and disable Checkstyle 
for now and beam will build/test within Eclipse.   I'm working on figuring out 
the checkstyle parts, but I wanted to submit this for review as this involves 
actual code changes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam eclipse-ide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/949.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #949


commit aa5495863fe2b87b61637ceeb278268bc69557a5
Author: Daniel Kulp <dk...@apache.org>
Date:   2016-09-12T21:31:22Z

Update code to import into eclipse using m2e and compile (checkstyle still 
generates a ton of errors)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---