Re: Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2195

2016-12-21 Thread Dan Halperin 
This was a manual invocation, which somehow causes an email to the commits
list even though it was not run against master. Please ignore.

Context: [BEAM-1194]

On Wed, Dec 21, 2016 at 11:41 AM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  MavenInstall/2195/changes>
>
>

Re: Build failed in Jenkins: beam_PostCommit_MavenVerify #1823

2016-11-15 Thread Dan Halperin 
Not your fault -- merge race and stale precommit results. Thanks for
solving the mystery!

More details:

* Added gridfs to mongodb-io
https://github.com/apache/incubator-beam/pull/1324
* Fixed findbugs for a stale version of mongodb-io
https://github.com/apache/incubator-beam/pull/1356

This is why we want a submit queue / merge bot :)

Dan

On Tue, Nov 15, 2016 at 3:36 PM, Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:

> Actually, the problem comes from the merge of MongoDB GridFS write.
>
> I'm fixing it.
>
> Sorry again.
>
> Regards
> JB
>
> On 11/15/2016 03:27 PM, Dan Halperin wrote:
>
>> Hmm, looks like heck
>> that https://github.com/apache/incubator-beam/pull/1356
>> <https://github.com/apache/incubator-beam/pull/1356> broke this despite
>> green precommit. Rolling back and investigating.
>>
>> On Tue, Nov 15, 2016 at 1:50 PM, Apache Jenkins Server
>> <jenk...@builds.apache.org <mailto:jenk...@builds.apache.org>> wrote:
>>
>> See
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> 1823/changes
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> 1823/changes>>
>>
>> Changes:
>>
>> [ansela] [BEAM-762] Unify spark-runner EvaluationContext and
>>
>> [dhalperi] [BEAM-930] Fix findbugs and re-enable Maven plugin in
>> MongoDbIO
>>
>> --
>> [...truncated 4741 lines...]
>> [INFO]
>> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce) @
>> beam-sdks-java-io-mongodb ---
>> [INFO]
>> [INFO] --- maven-remote-resources-plugin:1.5:process
>> (process-resource-bundles) @ beam-sdks-java-io-mongodb ---
>> [INFO]
>> [INFO] --- maven-resources-plugin:2.7:resources (default-resources)
>> @ beam-sdks-java-io-mongodb ---
>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>> [INFO] skip non existing resourceDirectory
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/resources
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/resources>>
>> [INFO] Copying 3 resources
>> [INFO]
>> [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @
>> beam-sdks-java-io-mongodb ---
>> [INFO] Changes detected - recompiling the module!
>> [INFO] Compiling 3 source files to
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/target/classes
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/target/classes>>
>> [WARNING] bootstrap class path not set in conjunction with -source 1.7
>> [INFO]
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java>>:
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java>>
>> uses or overrides a deprecated API.
>> [INFO]
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java>>:
>> Recompile with -Xlint:deprecation for details.
>> [INFO]
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mongodb/MongoDbGridFSIO.java>>:
>> Some input files use unchecked or unsafe operations.
>> [INFO]
>> <https://builds.apache.org/job/beam_PostCommit_MavenVerify/
>> ws/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/
>> io/mon

Re: [jira] [Created] (BEAM-961) CountingInput could have starting number

2016-11-10 Thread Dan Halperin 
Why not support this in a follow-on pardo that shifts the range?

On Thu, Nov 10, 2016 at 1:22 PM, Kenneth Knowles (JIRA) 
wrote:

> Kenneth Knowles created BEAM-961:
> 
>
>  Summary: CountingInput could have starting number
>  Key: BEAM-961
>  URL: https://issues.apache.org/jira/browse/BEAM-961
>  Project: Beam
>   Issue Type: New Feature
>   Components: sdk-java-core
> Reporter: Kenneth Knowles
> Priority: Trivial
>
>
> TL;DR: Add {{startingAt}} to {{CountingInput}}.
>
> Right now you can have {{CountingInput.upTo(someNumber)}} but it came up
> in a test that if you want to have, say, one PCollection that is 1 through
> 10 and another that is 11 through 20 - so you know they are disjoint - then
> it requires some boilerplate to add 10 to every element. That boilerplate
> should be part of the {{CountingInput}}
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Re: incubator-beam git commit: fixup! spark pom.xml: limit parallelism in runnable-on-service tests

2016-11-02 Thread Dan Halperin 
(iterating with Amit on the Spark RunnableOnService suites on this branch)

On Wed, Nov 2, 2016 at 9:34 AM, <dhalp...@apache.org> wrote:

> Repository: incubator-beam
> Updated Branches:
>   refs/heads/spark-ros 6e1652a06 -> f2637d745
>
>
> fixup! spark pom.xml: limit parallelism in runnable-on-service tests
>
>
> Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
> Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/
> commit/f2637d74
> Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f2637d74
> Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f2637d74
>
> Branch: refs/heads/spark-ros
> Commit: f2637d74500bb33e6393ea446d49f2591dbe7632
> Parents: 6e1652a
> Author: Dan Halperin <dhalp...@google.com>
> Authored: Wed Nov 2 09:33:28 2016 -0700
> Committer: Dan Halperin <dhalp...@google.com>
> Committed: Wed Nov 2 09:33:28 2016 -0700
>
> --
>  runners/spark/pom.xml | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> --
>
>
> http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/
> f2637d74/runners/spark/pom.xml
> --
> diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
> index 0f2bbec..d16bd0b 100644
> --- a/runners/spark/pom.xml
> +++ b/runners/spark/pom.xml
> @@ -73,7 +73,8 @@
>  
>org.apache.beam.sdk.testing.RunnableOnService groups>
>1
> -  4
> +  none
> +  false
>false
>true
>
>
>

Re: Build failed in Jenkins: beam_PostCommit_RunnableOnService_SparkLocal #27

2016-10-17 Thread Dan Halperin 
Sorry folks for the spam -- this is me playing around with the Spark
postcommit seeing if it somehow magically passes after 2 months of idleness.

I'll remove the email hook until I get it configured correctly.

On Mon, Oct 17, 2016 at 2:49 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  RunnableOnService_SparkLocal/27/changes>
>
> Changes:
>
> [lcwik] Move the step output ids to use a flat namespace. Also add a
> logical
>
> [dhalperi] Remove Remaining Nested Contexts from NullableCoder
>
> [bchambers] Fix SplittableParDoTest
>
> [dhalperi] AvroIO.Write: minor doc and test fixes
>
> --
> [...truncated 5861 lines...]
>   EmptyStreamAssertionTest.testFixedWindows:67 » Runtime
> java.net.BindException:...
>   FlattenStreamingTest.testFlattenBoundedUnbounded:99 » Runtime
> java.net.BindExc...
>   FlattenStreamingTest.testFlattenUnbounded:79 » Runtime
> java.net.BindException:...
>   KafkaStreamingTest.testRun:109 » Runtime java.net.BindException: Cannot
> assign...
>   ResumeFromCheckpointStreamingTest.testRun:120->run:160 » Runtime
> java.net.Bind...
>   SimpleStreamingWordCountTest.testFixedWindows:84 » Runtime
> java.net.BindExcept...
>
> Tests run: 40, Failures: 0, Errors: 26, Skipped: 0
>
> [ERROR] There are test failures.
>
> Please refer to  RunnableOnService_SparkLocal/ws/runners/spark/target/surefire-reports>
> for the individual test results.
> [JENKINS] Recording test results
> [INFO]
> [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ beam-runners-spark ---
> [INFO] Building jar:  RunnableOnService_SparkLocal/ws/runners/spark/target/beam-
> runners-spark-0.3.0-incubating-SNAPSHOT.jar>
> [INFO]
> [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @
> beam-runners-spark ---
> [INFO]
> [INFO] --- maven-jar-plugin:2.5:test-jar (default-test-jar) @
> beam-runners-spark ---
> [INFO] Building jar:  RunnableOnService_SparkLocal/ws/runners/spark/target/beam-
> runners-spark-0.3.0-incubating-SNAPSHOT-tests.jar>
> [INFO]
> [INFO] --- maven-shade-plugin:2.4.1:shade (default) @ beam-runners-spark
> ---
> [INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded
> jar.
> [INFO] Including org.xerial.snappy:snappy-java:jar:1.1.2.1 in the shaded
> jar.
> [INFO] Including commons-codec:commons-codec:jar:1.4 in the shaded jar.
> [INFO] Including org.mortbay.jetty:jetty:jar:6.1.26 in the shaded jar.
> [INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded
> jar.
> [INFO] Including commons-logging:commons-logging:jar:1.1.1 in the shaded
> jar.
> [INFO] Including commons-lang:commons-lang:jar:2.5 in the shaded jar.
> [INFO] Including commons-collections:commons-collections:jar:3.2.1 in the
> shaded jar.
> [INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.8.8 in the
> shaded jar.
> [INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 in the
> shaded jar.
> [INFO] Including com.google.protobuf:protobuf-java:jar:3.0.0 in the
> shaded jar.
> [INFO] Including io.netty:netty:jar:3.6.2.Final in the shaded jar.
> [INFO] Including com.google.guava:guava:jar:19.0 in the shaded jar.
> [INFO] Including com.google.auto.service:auto-service:jar:1.0-rc2 in the
> shaded jar.
> [INFO] Including com.google.auto:auto-common:jar:0.3 in the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.7.2 in the
> shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2
> in the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.7.2 in
> the shaded jar.
> [INFO] Including org.apache.avro:avro:jar:1.8.1 in the shaded jar.
> [INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.7 in the
> shaded jar.
> [INFO] Including org.tukaani:xz:jar:1.5 in the shaded jar.
> [INFO] Including org.slf4j:slf4j-api:jar:1.7.14 in the shaded jar.
> [INFO] Including joda-time:joda-time:jar:2.4 in the shaded jar.
> [INFO] Including 
> org.apache.beam:beam-sdks-java-core:jar:0.3.0-incubating-SNAPSHOT
> in the shaded jar.
> [INFO] Including io.grpc:grpc-auth:jar:1.0.1 in the shaded jar.
> [INFO] Including com.google.auth:google-auth-library-credentials:jar:0.4.0
> in the shaded jar.
> [INFO] Including io.grpc:grpc-core:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-context:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-netty:jar:1.0.1 in the shaded jar.
> [INFO] Including io.netty:netty-codec-http2:jar:4.1.3.Final in the shaded
> jar.
> [INFO] Including io.netty:netty-codec-http:jar:4.1.3.Final in the shaded
> jar.
> [INFO] Including io.grpc:grpc-stub:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-all:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-protobuf:jar:1.0.1 in the shaded jar.
> [INFO]

Re: Jenkins build became unstable: beam_PostCommit_MavenVerify #1525

2016-10-13 Thread Dan Halperin 
Filed https://issues.apache.org/jira/browse/BEAM-747

On Thu, Oct 13, 2016 at 5:33 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
>

Re: Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1503

2016-10-12 Thread Dan Halperin 
Just an FYI that the issues here were legitimate issues in an external
service that have since been resolved. They were present for approximately
90 minutes in a small set of places, and we were affected :)

On Tue, Oct 11, 2016 at 7:37 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  MavenVerify/org.apache.beam$beam-examples-java/1503/>
>
>

Re: Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1151

2016-09-15 Thread Dan Halperin 
(filed https://issues.apache.org/jira/browse/BEAM-632)

On Thu, Sep 15, 2016 at 12:14 AM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  GoogleCloudDataflow/changes>
>
>

Re: Build failed in Jenkins: beam_PostCommit_RunnableOnService_GearpumpLocal #146

2016-08-26 Thread Dan Halperin 
Sorry for the build spam everyone.

I filed BEAM-598 about this, and I was able to fixed compilation but then I
broken the runnable on service test suite. I think we need Manu's
intervention to fix.

For now, I've disabled the build.

On Fri, Aug 26, 2016 at 5:53 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  GearpumpLocal/146/changes>
>
> Changes:
>
> [dhalperi] gearpump: switch to stable version
>
> --
> [...truncated 17829 lines...]
>   PAssertTest.testContainsInAnyOrder:310 » Runtime Failed to construct
> instance ...
>   PAssertTest.testContainsInAnyOrderFalse:367 » Runtime Failed to
> construct inst...
>   PAssertTest.testContainsInAnyOrderNotSerializable:119 » Runtime Failed
> to cons...
>   PAssertTest.testGlobalWindowContainsInAnyOrder:322 » Runtime Failed to
> constru...
>   PAssertTest.testIsEqualTo:266 » Runtime Failed to construct instance
> from fact...
>   PAssertTest.testNotEqualTo:298 » Runtime Failed to construct instance
> from fac...
>   PAssertTest.testSerializablePredicate:142 » Runtime Failed to construct
> instan...
>   PAssertTest.testWindowedContainsInAnyOrder:334 » Runtime Failed to
> construct i...
>   PAssertTest.testWindowedIsEqualTo:278 » Runtime Failed to construct
> instance f...
>   PAssertTest.testWindowedSerializablePredicate:169 » Runtime Failed to
> construc...
>   ApproximateUniqueTest.testApproximateUniqueWithSmallInput:74 » Runtime
> Failed ...
>   CombineFnsTest.testComposedCombine:130 » Runtime Failed to construct
> instance ...
>   CombineFnsTest.testComposedCombineNullValues:247 » Runtime Failed to
> construct...
>   CombineFnsTest.testComposedCombineWithContext:185 » Runtime Failed to
> construc...
>   CombineTest.testAccumulatingCombine:496->runTestAccumulatingCombine:250
> » Runtime
>   CombineTest.testAccumulatingCombineEmpty:503->runTestAccumulatingCombine:250
> » Runtime
>   CombineTest.testBasicCombine:235->runTestBasicCombine:213 » Runtime
> Failed to ...
>   CombineTest.testBasicCombineEmpty:244->runTestBasicCombine:213 »
> Runtime Faile...
>   CombineTest.testCombineGloballyAsSingletonView:649 » Runtime Failed to
> constru...
>   CombineTest.testCombinePerKeyPrimitiveDisplayData:726 » Runtime Failed
> to cons...
>   CombineTest.testFixedWindowsCombine:270 » Runtime Failed to construct
> instance...
>   CombineTest.testFixedWindowsCombineWithContext:297 » Runtime Failed to
> constru...
>   CombineTest.testGlobalCombineWithDefaultsAndTriggers:385 » Runtime
> Failed to c...
>   CombineTest.testHotKeyCombining:540 » Runtime Failed to construct
> instance fro...
>   CombineTest.testHotKeyCombiningWithAccumulationMode:575 » Runtime
> Failed to co...
>   CombineTest.testSessionsCombine:412 » Runtime Failed to construct
> instance fro...
>   CombineTest.testSessionsCombineWithContext:438 » Runtime Failed to
> construct i...
>   CombineTest.testSimpleCombine:182->runTestSimpleCombine:131 » Runtime
> Failed t...
>   CombineTest.testSimpleCombineEmpty:206->runTestSimpleCombine:131 »
> Runtime Fai...
>   CombineTest.testSimpleCombineWithContext:190->
> runTestSimpleCombineWithContext:152 » Runtime
>   CombineTest.testSimpleCombineWithContextEmpty:199->
> runTestSimpleCombineWithContext:152 » Runtime
>   CombineTest.testSlidingWindowsCombineWithContext:335 » Runtime Failed
> to const...
>   CombineTest.testWindowedCombineEmpty:481 » Runtime Failed to construct
> instanc...
>   CountTest.testCountGloballyBasic:91 » Runtime Failed to construct
> instance fro...
>   CountTest.testCountGloballyEmpty:106 » Runtime Failed to construct
> instance fr...
>   CountTest.testCountPerElementBasic:55 » Runtime Failed to construct
> instance f...
>   CountTest.testCountPerElementEmpty:77 » Runtime Failed to construct
> instance f...
>   CreateTest.testCreate:87 » Runtime Failed to construct instance from
> factory m...
>   CreateTest.testCreateEmpty:100 » Runtime Failed to construct instance
> from fac...
>   CreateTest.testCreateTimestamped:242 » Runtime Failed to construct
> instance fr...
>   CreateTest.testCreateTimestampedEmpty:261 » Runtime Failed to construct
> instan...
>   CreateTest.testCreateWithKVVoidType:311 » Runtime Failed to construct
> instance...
>   CreateTest.testCreateWithNullsAndValues:144 » Runtime Failed to
> construct inst...
>   CreateTest.testCreateWithUnserializableElements:223 » Runtime Failed to
> constr...
>   CreateTest.testCreateWithVoidType:302 » Runtime Failed to construct
> instance f...
>   FilterTest.testFilterByPredicate:92 » Runtime Failed to construct
> instance fro...
>   FilterTest.testFilterGreaterThan:118 » Runtime Failed to construct
> instance fr...
>   FilterTest.testFilterGreaterThanEq:144 » Runtime Failed to construct
> instance ...
>   FilterTest.testFilterLessThan:105 » Runtime Failed to construct
> instance from ...
>   FilterTest.testFilterLessThanEq:131 » Runtime Failed to construct
> instance

Re: Build failed in Jenkins: beam_PostCommit_PythonVerify #67

2016-07-11 Thread Dan Halperin 
Filed JIRA: https://issues.apache.org/jira/browse/BEAM-441

On Mon, Jul 11, 2016 at 5:54 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  >
>
> Changes:
>
> [dhalperi] Fix warnings that came with newly released Pylint (1.6.1)
> version.
>
> --
> [...truncated 1247 lines...]
> echo ">>> RUNNING DIRECT RUNNER py-wordcount"
> >>> RUNNING DIRECT RUNNER py-wordcount
> python -m apache_beam.examples.wordcount --output /tmp/py-wordcount-direct
> INFO:root:Missing pipeline option (runner). Executing pipeline using the
> default runner: DirectPipelineRunner.
> INFO:oauth2client.client:Attempting refresh to obtain initial access_token
> INFO:root:Starting finalize_write threads with num_shards: 1, num_threads:
> 1
> INFO:root:Renamed 1 shards in 0.02 seconds.
> INFO:root:Final: Debug counters: {'element_counts':
> Counter({('group/reify_windows', None): 28001, ('pair_with_one', None):
> 28001, ('split', None): 28001, 'read': 5525, ('group/group_by_window',
> None): 4784, ('format', None): 4784, ('count', None): 4784,
> 'group/group_by_key': 4784, ('write/WriteImpl/finalize_write', None): 1,
> ('write/WriteImpl/write_bundles', None): 1,
> ('write/WriteImpl/initialize_write', None): 1, 'write/WriteImpl/DoOnce':
> 1})}
> INFO:root:number of empty lines: 1663
> INFO:root:average word lengths: [4.204242705617657]
> # TODO: check that output file is generated for Direct Runner.
>
> # Run wordcount on the service.
>
> # Where to store wordcount output.
> GCS_LOCATION=gs://temp-storage-for-end-to-end-tests
>
> # Job name needs to be unique
> JOBNAME=py-wordcount-`date +%s`
> date +%s
>
> PROJECT=apache-beam-testing
>
> # Create a tarball
> python setup.py sdist
> running sdist
> running egg_info
> writing requirements to apache_beam_sdk.egg-info/requires.txt
> writing apache_beam_sdk.egg-info/PKG-INFO
> writing top-level names to apache_beam_sdk.egg-info/top_level.txt
> writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
> reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
> writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
> warning: sdist: standard file not found: should have one of README,
> README.rst, README.txt
>
> running check
> warning: check: missing meta-data: if 'author' supplied, 'author_email'
> must be supplied too
>
> creating apache-beam-sdk-0.3.0
> creating apache-beam-sdk-0.3.0/apache_beam
> creating apache-beam-sdk-0.3.0/apache_beam/coders
> creating apache-beam-sdk-0.3.0/apache_beam/examples
> creating apache-beam-sdk-0.3.0/apache_beam/internal
> creating apache-beam-sdk-0.3.0/apache_beam/internal/clients
> creating apache-beam-sdk-0.3.0/apache_beam/internal/clients/bigquery
> creating apache-beam-sdk-0.3.0/apache_beam/internal/clients/dataflow
> creating apache-beam-sdk-0.3.0/apache_beam/internal/clients/storage
> creating apache-beam-sdk-0.3.0/apache_beam/io
> creating apache-beam-sdk-0.3.0/apache_beam/runners
> creating apache-beam-sdk-0.3.0/apache_beam/transforms
> creating apache-beam-sdk-0.3.0/apache_beam/typehints
> creating apache-beam-sdk-0.3.0/apache_beam/utils
> creating apache-beam-sdk-0.3.0/apache_beam_sdk.egg-info
> making hard links in apache-beam-sdk-0.3.0...
> hard linking setup.cfg -> apache-beam-sdk-0.3.0
> hard linking setup.py -> apache-beam-sdk-0.3.0
> hard linking apache_beam/__init__.py -> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/dataflow_test.py ->
> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/error.py -> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/pipeline.py -> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/pipeline_test.py ->
> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/pvalue.py -> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/pvalue_test.py ->
> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/version.py -> apache-beam-sdk-0.3.0/apache_beam
> hard linking apache_beam/coders/__init__.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/coder_impl.pxd ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/coder_impl.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/coders.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/coders_test.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/coders_test_common.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/fast_coders_test.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/observable.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/observable_test.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard linking apache_beam/coders/slow_coders_test.py ->
> apache-beam-sdk-0.3.0/apache_beam/coders
> hard

Re: Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #462

2016-06-01 Thread Dan Halperin 
Looks like this was a transient failure in GCS (expected on occasion) that
could/would have been resolved when the work item was retried. However, it
looks like the test framework canceled the pipeline before that task was
retried. Is this the desired behavior?

Thanks,
Dan

On Wed, Jun 1, 2016 at 7:24 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/462/changes
> >
>
>

Re: Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #384

2016-05-20 Thread Dan Halperin 
We're looking into these tests that have been failing for two days now.
Luckily, they're not on the pre-commit path so they're not blocking work.

On Fri, May 20, 2016 at 1:27 AM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/changes
> >
>
>

Re: Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #267

2016-05-03 Thread Dan Halperin 
The first one of these failed because of a flake. This time, it looks like
a test job was canceled? [link

]

2016-05-03T23:56:17.307Z: Basic:  (60e0e3f1fd152c6c): Cancel request
is committed for workflow job:
2016-05-03_16_54_54-13629073328441817237.


Davor? Jason?

Thanks,
Dan

On Tue, May 3, 2016 at 5:14 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/changes
> >
>
>

Re: Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #225

2016-04-26 Thread Dan Halperin 
This should be fixed now by a configuration change -- we recently made a
package rename, and the config of this Jenkins build was not updated.

On Tue, Apr 26, 2016 at 6:32 PM, Dan Halperin <dhalp...@google.com> wrote:

> Looking
>
> On Tue, Apr 26, 2016 at 6:21 PM, Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> See <
>> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/changes
>> >
>>
>>
>

Re: Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #225

2016-04-26 Thread Dan Halperin 
Looking

On Tue, Apr 26, 2016 at 6:21 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/changes
> >
>
>

Re: [jira] [Commented] (BEAM-190) Dead-letter drop for bad BigQuery records

2016-04-13 Thread Dan Halperin 
I thought that we were under the impression that rather than losing data
it's likely better to update your pipeline to handle these?

On Wed, Apr 13, 2016 at 10:59 AM, Luke Cwik (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/BEAM-190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239701#comment-15239701
> ]
>
> Luke Cwik commented on BEAM-190:
> 
>
> I believe this can easily extend beyond BigQuery to having a dead letter
> feature for failing DoFns of any kind.
>
> > Dead-letter drop for bad BigQuery records
> > -
> >
> > Key: BEAM-190
> > URL: https://issues.apache.org/jira/browse/BEAM-190
> > Project: Beam
> >  Issue Type: Bug
> >  Components: runner-core
> >Reporter: Mark Shields
> >Assignee: Frances Perry
> >
> > If a BigQuery insert fails for data-specific rather than structural
> reasons (eg cannot parse a date) then the bundle will be retried
> indefinitely, first by BigQueryTableInserter.insertAll then by the overall
> production retry logic of the underlying runner.
> > Better would be to allow customer to specify a dead-letter store for
> records such as those so that overall processing can continue while bad
> records are quarantined.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Re: [jira] [Commented] (BEAM-147) Rename Aggregator to [P]Metric

2016-03-25 Thread Dan Halperin 
I'm concerned that we will rename it, then the new semantics we choose will
result in a second rename. I agree with Davor that we should pin down the
interface and semantics so we only do the rename once.
On Fri, Mar 25, 2016 at 10:28 Robert Bradshaw (JIRA) 
wrote:

>
> [
> https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212123#comment-15212123
> ]
>
> Robert Bradshaw commented on BEAM-147:
> --
>
> There is consensus that the name "Aggregators" is confusing, and it makes
> sense to rename it as part of the larger set of backwards incompatible
> changes rather than later. Thought the exact semantics of this features is
> still being pounded out, it seems well enough defined to give it a name at
> this point.
>
> > Rename Aggregator to [P]Metric
> > --
> >
> > Key: BEAM-147
> > URL: https://issues.apache.org/jira/browse/BEAM-147
> > Project: Beam
> >  Issue Type: Bug
> >  Components: beam-model, sdk-java-core, sdk-py
> >Reporter: Robert Bradshaw
> >Assignee: Frances Perry
> >
> > The name "Aggregator" is confusing.
> > See discussion at
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
> and
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
> . Exact name still being bikeshedded.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>