Re: Beam Dependency Check Report (2018-06-13)
se:hbase-shaded-server 1.2.6 2.0.0-alpha2 2017-05-29 > 2018-05-31 > org.apache.hive:hive-cli 2.1.0 3.0.0.3.0.0.3-2 2016-06-16 2018-05-21 > org.apache.hive:hive-common 2.1.0 3.0.0.3.0.0.3-2 2016-06-16 2018-05-21 > org.apache.hive:hive-exec 2.1.0 3.0.0.3.0.0.3-2 2016-06-16 2018-05-21 > org.apache.hive.hcatalog:hive-hcatalog-core 2.1.0 3.0.0.3.0.0.3-2 > 2016-06-16 2018-05-21 > org.apache.httpcomponents:httpasyncclient 4.1.2 4.1.3 2016-06-18 > 2017-02-05 > org.apache.httpcomponents:httpclient 4.5.2 4.5.5 2016-02-21 2018-01-18 > org.apache.httpcomponents:httpcore 4.4.5 4.4.9 2016-06-08 2018-01-11 > net.java.dev.javacc:javacc 4.0 7.0.3 2018-06-08 2017-11-06 > jline:jline 2.14.6 3.0.0.M1 2018-03-26 2018-06-08 > net.java.dev.jna:jna 4.1.0 4.5.1 2014-03-06 2017-12-27 > com.esotericsoftware.kryo:kryo 2.21 2.24.0 2013-02-27 2014-05-04 > io.dropwizard.metrics:metrics-core 3.1.2 4.1.0-rc2 2015-04-25 2018-05-03 > org.mongodb:mongo-java-driver 3.2.2 3.8.0-beta3 2016-02-15 2018-05-29 > io.netty:netty-all 4.1.17.Final 5.0.0.Alpha2 2017-11-08 2018-06-06 > io.grpc:protoc-gen-grpc-java 1.2.0 1.12.0 2017-03-15 2018-05-07 > org.apache.qpid:proton-j 0.13.1 0.27.1 2016-07-01 2018-04-25 > com.carrotsearch.randomizedtesting:randomizedtesting-runner 2.5.0 2.6.3 > 2017-01-23 2018-06-11 > org.scala-lang:scala-library 2.11.8 2.13.0-M4 2017-03-08 2018-05-14 > org.slf4j:slf4j-api 1.7.25 1.8.0-beta2 2017-03-16 2018-03-21 > org.slf4j:slf4j-jdk14 1.7.25 1.8.0-beta2 2017-03-16 2018-03-21 > org.apache.solr:solr-core 5.5.4 7.3.1 2017-10-20 2018-05-17 > org.apache.solr:solr-solrj 5.5.4 7.3.1 2017-10-20 2018-05-17 > org.apache.solr:solr-test-framework 5.5.4 7.3.1 2017-10-20 2018-05-17 > org.springframework:spring-expression 4.3.5.RELEASE 5.0.7.RELEASE > 2017-01-25 2018-06-12 > sqlline:sqlline 1.3.0 1.4.0 2017-05-30 2018-05-30 > com.clearspring.analytics:stream 2.9.5 2.9.6 2016-08-10 2018-01-10 > org.elasticsearch.client:transport 5.0.0 6.2.4 2016-10-25 2018-04-12 > org.elasticsearch.plugin:transport-netty4-client 5.6.3 6.2.4 2017-11-06 > 2018-04-12 > org.tukaani:xz 1.5 1.8 2014-03-08 2018-01-04 > > -- *Paul Gerver*
Re: TestPipeline drops options with JsonIgnore annotation
I am hitting this issue on several integration tests, but these tests are only using options set by the `beamTestPipelineOptions` system property and not going through convertToArgs from my side. I apologize for the email switch but the mail list isn’t letting log in through Google+. On 2018/01/22 21:14:15, Lukasz Cwik <l...@google.com> wrote: > Are your talking about integration tests that use a main like WordCountIT?> > > If so, then https://github.com/apache/beam/pull/4346 was an attempt to get> > around this limitation but I suggested that we get rid of convertToArgs> > completely so there is no serialization round trip before the args are> > passed to the TestPipeline.> > If you have some ideas in this space, feel free to contribute a PR.> > > On Fri, Jan 19, 2018 at 4:54 PM, Paul Gerver <pf...@gmail.com> wrote:> > > > Hello,> > >> > > With Beam 2.2 it looks like the TestPipeline now serializes options before> > > giving them to the parent Pipeline to run (in order to utilize runtime> > > options). I have some options that were marked with the `JsonIgnore`> > > annotation which now seem to be getting dropped for my runner.> > >> > > Is there something I'm missing which would allow me to skip this> > > serialization piece in the TestPipeline? If not, this seems like a side> > > effect of 2.2.> > >> > > Let me know!> > >> > > Thanks,> > > --> > > *Paul Gerver*> > >> > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
TestPipeline drops options with JsonIgnore annotation
Hello, With Beam 2.2 it looks like the TestPipeline now serializes options before giving them to the parent Pipeline to run (in order to utilize runtime options). I have some options that were marked with the `JsonIgnore` annotation which now seem to be getting dropped for my runner. Is there something I'm missing which would allow me to skip this serialization piece in the TestPipeline? If not, this seems like a side effect of 2.2. Let me know! Thanks, -- *Paul Gerver*
Re: Jira Access
Oh, yes. When I registered pgerv12 my browser said it timed out. I tried to register again, said it already existed so I created pfgerver. Unfortunately, I didn't see a way to delete the pfgerver account. Do you know if a note to the Jira admins could handle it? On 2017-11-08 18:02, Lukasz Cwik <l...@google.com.INVALID> wrote: > I have added you and saw the Jira that you commented on and assigned it to> > you.> > > Curious note, I also saw a pfgerver which also seems to be you.> > > On Wed, Nov 8, 2017 at 2:08 PM, Paul Gerver <pf...@gmail.com> wrote:> > > > Hello,> > >> > > I'm part of the IBM Streams team and would like to contribute to the Apache> > > Beam community.> > > My ASF Jira ID is pgerv12.> > >> > > Thanks!> > >> > > --> > >> > > *Paul Gerver*> > >> >
Jira Access
Hello, I'm part of the IBM Streams team and would like to contribute to the Apache Beam community. My ASF Jira ID is pgerv12. Thanks! -- *Paul Gerver*
Re: Combine.Global
Ah, I found my mistake. You overrode the getAccumulator and
getDefaultOutputCoders which my implementation did not. This approach is
straight forward now.
Thanks!
On 2017-04-07 23:46 (-0500), Aviem Zur <aviem...@gmail.com> wrote:
> I wasn't able to reproduce the issue you're experiencing.
> I've created a gist with an example that works and is similar to what you
> have described.
> Please help us make tweaks to the gist reproduce your problem:
> https://gist.github.com/aviemzur/ba213d98b4484492099b3cf709ddded0
>
> On Fri, Apr 7, 2017 at 7:25 PM Paul Gerver <pfger...@gmail.com> wrote:
>
> > Yes, the pipeline is quite small:
> >
> > pipeline.apply("source",
> > Read.from(new CustomSource())).setCoder(CustomSource.coder)
> > .apply("GlobalCombine", Combine.globally(new
> > CustomCombineFn())).setCoder(CustomTuple.coder);
> >
> >
> > The InputT is not the same as OutputT, so the input coder can't be used.
> >
> > On 2017-04-07 08:58 (-0500), Aviem Zur <aviem...@gmail.com> wrote:
> > > Have you set the coder for your input PCollection? The one on which you
> > > perform the Combine?
> > >
> > > On Fri, Apr 7, 2017 at 4:24 PM Paul Gerver <pfger...@gmail.com> wrote:
> > >
> > > > Hello All,
> > > >
> > > > I'm trying to test out a Combine.Globally transform which takes in a
> > small
> > > > custom class (CustomA) and outputs a secondary custom class (CustomB).
> > I
> > > > have set the coder for the resulting PCollection, but Beam is
> > > > arguing that a coder for a KV type is missing (see output at bottom).
> > > >
> > > > Since this a global combine, the input nor the output is of KV type,
> > so I
> > > > decided to take a look at the Combine code. Since
> > Combine.Globally.expand()
> > > > performs a perKeys and groupedValues underneath the covers, but
> > requires
> > > > making an intermediate PCollection KV<Void, OutputT> which--according
> > to
> > > > the docs--is inferred from the CombineFn.
> > > >
> > > > I believe I could workaround this by registering a KvCoder with the
> > > > CoderRegistry, but that's not intuitive. Is there a better way to
> > address
> > > > this currently, or should something be added to the CombineFn area for
> > > > setting an output coder similar to PCollection.
> > > >
> > > >
> > > > Output:
> > > > Exception in thread "main" java.lang.IllegalStateException: Unable to
> > > > return a default Coder for
> > > >
> > > >
> > GlobalCombine/Combine.perKey(CustomTuple)/Combine.GroupedValues/ParDo(Anonymous).out
> > > > [Class]. Correct one of the following root causes:
> > > > No Coder has been manually specified; you may do so using
> > .setCoder().
> > > > Inferring a Coder from the CoderRegistry failed: Unable to provide a
> > > > default Coder for org.apache.beam.sdk.values.KV<K, OutputT>. Correct
> > one of
> > > > the following root causes:
> > > > Building a Coder using a registered CoderFactory failed: Cannot
> > provide
> > > > coder for parameterized type org.apache.beam.sdk.values.KV<K, OutputT>:
> > > > Unable to provide a default Coder for java.lang.Object. Correct one of
> > the
> > > > following root causes:
> > > >
> > > >
> > > > Stack:
> > > > at
> > > >
> > > >
> > org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:174)
> > > > at
> > > > org.apache.beam.sdk.values.TypedPValue.getCoder(TypedPValue.java:51)
> > > > at
> > > > org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:130)
> > > > at
> > > >
> > > >
> > org.apache.beam.sdk.values.TypedPValue.finishSpecifying(TypedPValue.java:90)
> > > > at
> > > >
> > > >
> > org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:95)
> > > > at
> > org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:386)
> > > > at
> > org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:302)
> > > > at
> > > > org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
> > > > at
> > > >
> > org.apache.beam.sdk.transforms.Combine$Globally.expand(Combine.java:1460)
> > > > at
> > > >
> > org.apache.beam.sdk.transforms.Combine$Globally.expand(Combine.java:1337)
> > > > at
> > > >
> > org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
> > > > at
> > > >
> > org.apache.beam.runners.direct.DirectRunner.apply(DirectRunner.java:296)
> > > > at
> > org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:388)
> > > > at
> > org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:318)
> > > > at
> > > > org.apache.beam.sdk.values.PCollection.apply(PCollection.java:167)
> > > > at
> > > > org.iastate.edu.CombineTestPipeline.main(CombineTestPipeline.java:110)
> > > >
> > > >
> > > > Let me know. Thanks!
> > > > -Paul G
> > > >
> > > > --
> > > > *Paul Gerver*
> > > > pfger...@gmail.com
> > > >
> > >
> >
>
Combine.Global
Hello All, I'm trying to test out a Combine.Globally transform which takes in a small custom class (CustomA) and outputs a secondary custom class (CustomB). I have set the coder for the resulting PCollection, but Beam is arguing that a coder for a KV type is missing (see output at bottom). Since this a global combine, the input nor the output is of KV type, so I decided to take a look at the Combine code. Since Combine.Globally.expand() performs a perKeys and groupedValues underneath the covers, but requires making an intermediate PCollection KV<Void, OutputT> which--according to the docs--is inferred from the CombineFn. I believe I could workaround this by registering a KvCoder with the CoderRegistry, but that's not intuitive. Is there a better way to address this currently, or should something be added to the CombineFn area for setting an output coder similar to PCollection. Output: Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for GlobalCombine/Combine.perKey(CustomTuple)/Combine.GroupedValues/ParDo(Anonymous).out [Class]. Correct one of the following root causes: No Coder has been manually specified; you may do so using .setCoder(). Inferring a Coder from the CoderRegistry failed: Unable to provide a default Coder for org.apache.beam.sdk.values.KV<K, OutputT>. Correct one of the following root causes: Building a Coder using a registered CoderFactory failed: Cannot provide coder for parameterized type org.apache.beam.sdk.values.KV<K, OutputT>: Unable to provide a default Coder for java.lang.Object. Correct one of the following root causes: Stack: at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:174) at org.apache.beam.sdk.values.TypedPValue.getCoder(TypedPValue.java:51) at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:130) at org.apache.beam.sdk.values.TypedPValue.finishSpecifying(TypedPValue.java:90) at org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:95) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:386) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:302) at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154) at org.apache.beam.sdk.transforms.Combine$Globally.expand(Combine.java:1460) at org.apache.beam.sdk.transforms.Combine$Globally.expand(Combine.java:1337) at org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76) at org.apache.beam.runners.direct.DirectRunner.apply(DirectRunner.java:296) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:388) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:318) at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:167) at org.iastate.edu.CombineTestPipeline.main(CombineTestPipeline.java:110) Let me know. Thanks! -Paul G -- *Paul Gerver* pfger...@gmail.com
