Repository: incubator-beam
Updated Branches:
refs/heads/master a17a99f58 -> baf5e416d
[BEAM-592] Fix SparkRunner Dependency Problem in WordCount
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ef828de6
[BEAM-579] Integrate NamedAggregators into Spark sink system
This closes #867
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f346c877
Tree:
Repository: incubator-beam
Updated Branches:
refs/heads/master 204678323 -> f346c877a
Added support for reporting aggregator values to Spark sinks
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/226dea2f
Repository: incubator-beam
Updated Branches:
refs/heads/master 0ab9495f8 -> fc87a0ca7
SparkRunner batch interval as a configuration instead of Beam Windows.
Add the batch interval to the pipeline options, default arbitrarily to 1000
msec.
Pick-up the batch interval from pipeline options and
[BEAM-542] Spark batch interval should be a configuration instead of an
interpretation of the Pipeline's windows
This closes #808
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fc87a0ca
Tree:
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/743
[BEAM-492] Spark runner should call Pipeline.run() instead of
SparkRunner.run()
Be sure to do all of the following to help us incorporate your contribution
quickly and easily
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/736
[BEAM-491]-Reuse context and disable UI in the Spark runner tests
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make
Github user amitsela closed the pull request at:
https://github.com/apache/incubator-beam/pull/620
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/620
[BEAM-434] When examples write output to file it creates many output files
instead of one
Be sure to do all of the following to help us incorporate your contribution
quickly and easily
Github user amitsela closed the pull request at:
https://github.com/apache/incubator-beam/pull/585
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/585
Merge master into runners-spark2 branch to keep up-to-date
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure
Github user amitsela closed the pull request at:
https://github.com/apache/incubator-beam/pull/539
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user amitsela reopened a pull request:
https://github.com/apache/incubator-beam/pull/539
[BEAM-380] Remove Spark runner dependency on beam-examlpes-java
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [x] Make
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/539
[BEAM-380] Remove Spark runner dependency on beam-examlpes-java
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/495
Support spark-2.0
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[BEAM
Repository: incubator-beam
Updated Branches:
refs/heads/runners-spark2 [created] f57e66c48
[BEAM-305] Replace usages of PCollection.setCoder with Create.of().withCoder in
Spark Runner
This closes #386
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9f97ea0a
Tree:
Repository: incubator-beam
Updated Branches:
refs/heads/master a3fc40aa3 -> 9f97ea0a7
Use Create.of withCoder instead of setCoder on the created PCollection
One more left..
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit:
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/331
VoidCoder doesn't get special treatment in create() for streaming anyâ¦
Be sure to do all of the following to help us incorporate your contribution
quickly and easily
Update README according to dataflow->beam package rename
Annotate class with RunWith and some whitespace fix ups
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7fd9e1e7
Tree:
[BEAM-213] Fix README to use refactored package names and use AutoService for
Registrar
This closes #230
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/692f3a13
Tree:
Add a unit test for the SparkRunnerRegistrar
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f424b8d8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f424b8d8
Diff:
Repository: incubator-beam
Updated Branches:
refs/heads/master cf4c3e204 -> 692f3a136
Remove specific registrar classes and service files
Add dependency on AutoService
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit:
Github user amitsela closed the pull request at:
https://github.com/apache/incubator-beam/pull/226
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/226
[BEAM-213] Fix README to use refactored package names
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR
Repository: incubator-beam
Updated Branches:
refs/heads/master f20bf8afd -> 135cb733f
Materialize PCollection/RDD as windowed values with the appropriate windows.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit:
Replace valueInEmptyWindows with valueInGlobalWindow
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5fab1c5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5fab1c5f
Diff:
Replace valueInEmptyWindows with valueInGlobalWindow in Spark Function, and add
per-value (non-RDD)
windowing functions
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d852c5b9
Tree:
Repository: incubator-beam
Updated Branches:
refs/heads/master d42a07086 -> 1cef64261
[BEAM-43] Upgrade to Spark 1.6.1
Replace deprecated Function with VoidFunction
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit:
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/159
[Beam-43] Upgrade to Spark 1.6
Upgrade to Spark 1.6.1 (latest) and resolve some involved deprecations.
You can merge this pull request into a Git repository by running:
$ git pull
Repository: incubator-beam
Updated Branches:
refs/heads/master 49d82baf1 -> 706fc5376
[BEAM-109] fix support for FixedWindows and SlidingWindows in batch
[BEAM-109] Better testing for FixedWindows and SlidingWindows
[BEAM-109] lower counts is unordered so better to compare entire result and
[BEAM-109] Combine.PerKey ignores grouping also by windows
This closes #63
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/706fc537
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/706fc537
Repository: incubator-beam
Updated Branches:
refs/heads/master ef1e32dee -> 659f0b877
[BEAM-113] Update Spark runner README
Just a couple of more changes
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit:
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/55
[Beam 113] Update Spark runner README
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amitsela/incubator-beam BEAM-113
Alternatively
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/63
[Beam 109] Combine.PerKey ignores grouping also by windows
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amitsela/incubator-beam BEAM
This closes #55
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/659f0b87
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/659f0b87
Diff:
[BEAM-11] second iteration of package reorganisation
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/eb0341d4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/eb0341d4
Diff:
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb0341d4/runners/spark/src/test/java/org/apache/beam/runners/spark/streaming/KafkaStreamingTest.java
--
diff --git
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb0341d4/runners/spark/src/main/java/org/apache/beam/runners/spark/TransformTranslator.java
--
diff --git
[BEAM-11] Add Spark runner to runners module
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bde9933d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bde9933d
Diff:
[BEAM-11] add Spark runner to included runners
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b49e3c95
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b49e3c95
Diff:
[BEAM-11] Replaced license headers to ASF license
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6ef36411
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6ef36411
Diff:
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb0341d4/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
--
diff --git
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/41c4ca6a/runners/spark/src/test/java/com/cloudera/dataflow/spark/NumShardsTest.java
--
diff --git
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/41c4ca6a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/hadoop/ShardNameTemplateHelper.java
--
diff --git
[BEAM-11] Spark runner directory structure and pom setup.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/41c4ca6a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/41c4ca6a
Diff:
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/41c4ca6a/runners/spark/src/main/java/com/cloudera/dataflow/spark/ShardNameTemplateHelper.java
--
diff --git
[BEAM-11] relocate Guava used by Dataflow (v19) since it conflicts with version
used by Hadoop (v11)
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/95ebf890
Tree:
Update README to latest version (0.4.0).
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/922508c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/922508c0
Diff:
Set the RDD's name from the PValue's name, to help diagnosis.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5069eedb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5069eedb
Diff:
[maven-release-plugin] prepare release spark-dataflow-0.3.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d7a35bdf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d7a35bdf
Diff:
Add spark-streaming support to spark-dataflow
Add support for application name and streaming (default: false)
Add pipeline options for streaming
Add print output as an unbounded write
Add default window strategy to represent Spark streaming micro-batches as fixed
windows
This translator
Update README to latest version (0.2.2).
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/27349adc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/27349adc
Diff:
First wave of changes from feedback
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1229b009
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1229b009
Diff:
The example needs --inputFile, not --input, to designate the input file
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f930380b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f930380b
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7a2e9a72/runners/spark/src/main/java/com/cloudera/dataflow/spark/streaming/SparkStreamingPipelineOptions.java
--
diff --git
Add a system property, dataflow.spark.directBroadcast, to allow pipelines to
bypass coders for broadcasts.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2d00b3bd
Tree:
Avoid warning email by not running codecov unless it was configured; update
jacoco and shade plugins
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b1441f4
Tree:
Update README to latest version (0.4.1).
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4b98c163
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4b98c163
Diff:
[maven-release-plugin] prepare release spark-dataflow-0.2.3
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fe0b8e9a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fe0b8e9a
Diff:
Repository: incubator-beam
Updated Branches:
refs/heads/master 0442a2416 -> b2b5f429f
Implement getAggregatorValues.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/89e2bb52
Tree:
Add tests for Spark 1.4 / 1.5 in Travis
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/87797015
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/87797015
Diff:
[maven-release-plugin] prepare for next development iteration
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/72167a2c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/72167a2c
Diff:
Correct input parameter is --inputFile
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0c84c9d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0c84c9d7
Diff:
Add support for writes with HadoopIO. This allows Hadoop
FileOutputFormats to be used with Spark Dataflow, as long as they
implement the ShardNameTemplateAware interface. This is easily achieved
by subclassing the desired FileOutputFormat class, see
TemplatedSequenceFileOutputFormat for an
Prevent possible NPE.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/44158622
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/44158622
Diff:
Remove some HadoopIO.Read.Bound factory methods and fluent setters; always set
key/value at creation
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b47a8d0a
Tree:
Update README to latest version (0.4.2).
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90c49b4f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90c49b4f
Diff:
More cleanup. View.AsSingleton is already exercised by the TfIdf test.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/78d66145
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/78d66145
Diff:
Only accumulate outputs from one call to processContext, rather than
for the whole partition.
Fixes https://github.com/cloudera/spark-dataflow/issues/61.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit:
[maven-release-plugin] prepare release spark-dataflow-0.4.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5ec8d59c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5ec8d59c
Diff:
[maven-release-plugin] prepare release spark-dataflow-0.4.2
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b8949b81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b8949b81
Diff:
Fix bug where values written to the output in DoFn#startBundle and
DoFn#finishBundle
were being ignored. Introduced in 62830a0.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/76815589
Tree:
[maven-release-plugin] prepare release spark-dataflow-0.2.2
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ebf70534
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ebf70534
Diff:
[maven-release-plugin] prepare release spark-dataflow-0.4.1
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3e767f5a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3e767f5a
Diff:
[maven-release-plugin] prepare for next development iteration
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4ec8c606
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4ec8c606
Diff:
Update to dataflow 0.4.150727.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/89945bf6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/89945bf6
Diff:
101 - 177 of 177 matches
Mail list logo