Re: Request for review of PR [Beam-8564]

[Amogh Tiwari]

Apologies for a late reply. The PR has been updated now. Would love to get
your thoughts/suggestions.

On Tue, Jan 7, 2020 at 12:36 AM Luke Cwik  wrote:

> Have you had a chance to update the PR?
>
> On Mon, Dec 30, 2019 at 5:00 AM Amogh Tiwari  wrote:
>
>> Hi Luke,
>>
>> We have gone through shevek/lzo-java, but we chose to go with
>> airflow/aircompressor for the following reasons:
>>
>> 1) shevek/lzo-java is internally using .jni, .c and .h files, hence the
>> GNU licence, and that would leave us with only choice of putting this as an
>> option dependency
>>
>> 2) performance of airlift/aircompressor was much better as compared to
>> shevek/lzo-java in terms of compression ratio and time taken for
>> compression/decompression
>>
>> 3) airflow/aircompressor is in pure java and is under Apache licence
>>
>> Therefore, we'd prefer to go with adding the current implementation as
>> optional. We'd require your inputs on the same as we are unsure on where we
>> are supposed to keep the required files and how the final directory
>> structure would like. We have an idea and we'll update the current PR
>> accordingly.
>>
>> Please do guide us on this.
>>
>>
>> Regards,
>>
>> Amogh Tiwari
>>
>> On Wed, Dec 18, 2019 at 4:42 AM Luke Cwik  wrote:
>>
>>> Sorry for the long delay (was on vacation).
>>>
>>> Using org.apache.hadoop isn't part of the Apache Beam Core module but is
>>> a dependency for those who depend on the Apache Beam Hadoop module. So I
>>> don't think swapping the com.facebook.presto.hadoop version for the
>>> org.apache.hadoop version will address Ismael's concern about including
>>> hadoop dependencies as part of the core.
>>>
>>> I looked at shevek/lzo-java[1] and I think its our best choice since it
>>> is:
>>> * pure Java
>>> * GPLv3 (would require marking the dependency optional and telling users
>>> to add it explicitly which we have done in the past as well)
>>> * small (<100kb)
>>> * small dependency tree (commons-logging & findbugs annotations if we
>>> only depend on lzo-core)
>>> * performance (github page claims 500mb/s compression and 800mb/s
>>> decompression)
>>>
>>> Alternatively we can make the LZO compression an extension module (with
>>> the facebook dependency) and use a registrar to have it loaded dynamically.
>>>
>>> 1: https://github.com/shevek/lzo-java
>>>
>>> On Fri, Dec 6, 2019 at 5:09 AM Amogh Tiwari  wrote:
>>>
>>>> While studying the code, we found that the airlift/ aircompressor
>>>> library only requires some classes which are also present in apache hadoop
>>>> common package. Therefore, we are now thinking that if we make changes in
>>>> the airlift/ aircompressor package, remove the
>>>> com.facebook.presto.hadoop and use the existing org.apache.hadoop
>>>> <https://mvnrepository.com/artifact/org.apache.hadoop> package which
>>>> is already included in beam. This will solve both #2 and #3 as the
>>>> transitive dependency will be removed and the size will also be reduced by
>>>> almost ~20mbs.
>>>>
>>>> But if we use this approach, we will have to manually change the util
>>>> whenever any changes are made to the airlift library.
>>>>
>>>> On Wed, Dec 4, 2019 at 10:13 PM Luke Cwik  wrote:
>>>>
>>>>> Going with the Registrar/ServiceLoader route would allow for
>>>>> alternative providers for the same compression algorithms so if they don't
>>>>> like one they can always contribute a different one.
>>>>>
>>>>> On Wed, Dec 4, 2019 at 8:22 AM Ismaël Mejía  wrote:
>>>>>
>>>>>> (1) seems not to be the issue because it is Apache licensed.
>>>>>> (2) and (3) are the big issues, because it requires a provided huge
>>>>>> uber jar that essentially leaks Hadoop classes into core SDK [1] so it is
>>>>>> definitely concerning.
>>>>>>
>>>>>> We discussed at some point during the PR that added ZStandard support
>>>>>> about creating some sort of Registrar for compression algorithms [2] but 
>>>>>> we
>>>>>> decided to not go ahead because we could achieve that for the zstd case 
>>>>>> via
>>>>>> the optional dependencies of commons-compress. Maybe it is time to
>>>>>> r

Re: Request for review of PR [Beam-8564]

[Amogh Tiwari]

Hi Luke,

We have gone through shevek/lzo-java, but we chose to go with
airflow/aircompressor for the following reasons:

1) shevek/lzo-java is internally using .jni, .c and .h files, hence the GNU
licence, and that would leave us with only choice of putting this as an
option dependency

2) performance of airlift/aircompressor was much better as compared to
shevek/lzo-java in terms of compression ratio and time taken for
compression/decompression

3) airflow/aircompressor is in pure java and is under Apache licence

Therefore, we'd prefer to go with adding the current implementation as
optional. We'd require your inputs on the same as we are unsure on where we
are supposed to keep the required files and how the final directory
structure would like. We have an idea and we'll update the current PR
accordingly.

Please do guide us on this.


Regards,

Amogh Tiwari

On Wed, Dec 18, 2019 at 4:42 AM Luke Cwik  wrote:

> Sorry for the long delay (was on vacation).
>
> Using org.apache.hadoop isn't part of the Apache Beam Core module but is a
> dependency for those who depend on the Apache Beam Hadoop module. So I
> don't think swapping the com.facebook.presto.hadoop version for the
> org.apache.hadoop version will address Ismael's concern about including
> hadoop dependencies as part of the core.
>
> I looked at shevek/lzo-java[1] and I think its our best choice since it is:
> * pure Java
> * GPLv3 (would require marking the dependency optional and telling users
> to add it explicitly which we have done in the past as well)
> * small (<100kb)
> * small dependency tree (commons-logging & findbugs annotations if we only
> depend on lzo-core)
> * performance (github page claims 500mb/s compression and 800mb/s
> decompression)
>
> Alternatively we can make the LZO compression an extension module (with
> the facebook dependency) and use a registrar to have it loaded dynamically.
>
> 1: https://github.com/shevek/lzo-java
>
> On Fri, Dec 6, 2019 at 5:09 AM Amogh Tiwari  wrote:
>
>> While studying the code, we found that the airlift/ aircompressor library
>> only requires some classes which are also present in apache hadoop common
>> package. Therefore, we are now thinking that if we make changes in the
>> airlift/ aircompressor package, remove the
>> com.facebook.presto.hadoop and use the existing org.apache.hadoop
>> <https://mvnrepository.com/artifact/org.apache.hadoop> package which is
>> already included in beam. This will solve both #2 and #3 as the transitive
>> dependency will be removed and the size will also be reduced by almost
>> ~20mbs.
>>
>> But if we use this approach, we will have to manually change the util
>> whenever any changes are made to the airlift library.
>>
>> On Wed, Dec 4, 2019 at 10:13 PM Luke Cwik  wrote:
>>
>>> Going with the Registrar/ServiceLoader route would allow for alternative
>>> providers for the same compression algorithms so if they don't like one
>>> they can always contribute a different one.
>>>
>>> On Wed, Dec 4, 2019 at 8:22 AM Ismaël Mejía  wrote:
>>>
>>>> (1) seems not to be the issue because it is Apache licensed.
>>>> (2) and (3) are the big issues, because it requires a provided huge
>>>> uber jar that essentially leaks Hadoop classes into core SDK [1] so it is
>>>> definitely concerning.
>>>>
>>>> We discussed at some point during the PR that added ZStandard support
>>>> about creating some sort of Registrar for compression algorithms [2] but we
>>>> decided to not go ahead because we could achieve that for the zstd case via
>>>> the optional dependencies of commons-compress. Maybe it is time to
>>>> reconsider if such mechanism is worth. For example for users that may not
>>>> care about having the hadoop leakage to be able to use LZO.
>>>>
>>>> Refs.
>>>> [1] https://mvnrepository.com/artifact/io.airlift/aircompressor/0.16
>>>> [2] https://issues.apache.org/jira/browse/BEAM-6422
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Dec 3, 2019 at 7:01 PM Robert Bradshaw 
>>>> wrote:
>>>>
>>>>> Is there a way to wrap this up as an optional dependency with multiple
>>>>> possible providers, if there's no good library satisfying all of the
>>>>> conditions (in particular (1))?
>>>>>
>>>>> On Tue, Dec 3, 2019 at 9:47 AM Luke Cwik  wrote:
>>>>> >
>>>>> > I was hoping that someone in the community would provide some
>>>>> alternatives since there are quite a few impleme

Re: Request for review of PR [Beam-8564]

[Amogh Tiwari]

While studying the code, we found that the airlift/ aircompressor library
only requires some classes which are also present in apache hadoop common
package. Therefore, we are now thinking that if we make changes in the
airlift/ aircompressor package, remove the
com.facebook.presto.hadoop and use the existing org.apache.hadoop
<https://mvnrepository.com/artifact/org.apache.hadoop> package which is
already included in beam. This will solve both #2 and #3 as the transitive
dependency will be removed and the size will also be reduced by almost
~20mbs.

But if we use this approach, we will have to manually change the util
whenever any changes are made to the airlift library.

On Wed, Dec 4, 2019 at 10:13 PM Luke Cwik  wrote:

> Going with the Registrar/ServiceLoader route would allow for alternative
> providers for the same compression algorithms so if they don't like one
> they can always contribute a different one.
>
> On Wed, Dec 4, 2019 at 8:22 AM Ismaël Mejía  wrote:
>
>> (1) seems not to be the issue because it is Apache licensed.
>> (2) and (3) are the big issues, because it requires a provided huge uber
>> jar that essentially leaks Hadoop classes into core SDK [1] so it is
>> definitely concerning.
>>
>> We discussed at some point during the PR that added ZStandard support
>> about creating some sort of Registrar for compression algorithms [2] but we
>> decided to not go ahead because we could achieve that for the zstd case via
>> the optional dependencies of commons-compress. Maybe it is time to
>> reconsider if such mechanism is worth. For example for users that may not
>> care about having the hadoop leakage to be able to use LZO.
>>
>> Refs.
>> [1] https://mvnrepository.com/artifact/io.airlift/aircompressor/0.16
>> [2] https://issues.apache.org/jira/browse/BEAM-6422
>>
>>
>>
>>
>> On Tue, Dec 3, 2019 at 7:01 PM Robert Bradshaw 
>> wrote:
>>
>>> Is there a way to wrap this up as an optional dependency with multiple
>>> possible providers, if there's no good library satisfying all of the
>>> conditions (in particular (1))?
>>>
>>> On Tue, Dec 3, 2019 at 9:47 AM Luke Cwik  wrote:
>>> >
>>> > I was hoping that someone in the community would provide some
>>> alternatives since there are quite a few implementations.
>>> >
>>> > On Tue, Dec 3, 2019 at 8:20 AM Amogh Tiwari  wrote:
>>> >>
>>> >> Hi Luke,
>>> >>
>>> >> I agree with your thoughts and observations. But,
>>> airlift:aircompressor is the only implementation of LZO in pure java. That
>>> straight away solves #5.
>>> >> The other implementations that I found either have licensing issues
>>> (since LZO natively uses GNU GPL licence) or are implemented using .c, .h
>>> and jni (which again make them dependent on the OS). Please refer these:
>>> twitter/hadoop-lzo and shevek/lzo-java.
>>> >> These were the main reasons why we based this on
>>> airlift:aircompressor.
>>> >>
>>> >> Thanks and Regards,
>>> >> Amogh
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Dec 3, 2019 at 2:59 AM Luke Cwik  wrote:
>>> >>>
>>> >>> I took a look. My biggest concern is finding a good LZO
>>> implementation. Looking for one that preferably has:
>>> >>> 1) Apache license
>>> >>> 2) Has zero transitive dependencies
>>> >>> 3) Is small
>>> >>> 4) Is performant
>>> >>> 5) Is native java or supports execution on the three main OSs
>>> (Windows, Linux, Mac)
>>> >>>
>>> >>> In your PR you suggested using io.airlift:aircompressor:0.16 which
>>> doesn't meet item #2 and its transitive dependency fails #3.
>>> >>>
>>> >>> On Mon, Dec 2, 2019 at 12:16 PM Amogh Tiwari 
>>> wrote:
>>> >>>>
>>> >>>> Hi,
>>> >>>> I have filed a PR for an extension that will enable Apache Beam to
>>> work with LZO/LZOP compression. Please refer.
>>> >>>> I would love it if someone can take this up and review it.
>>> >>>> Please feel free to share your thoughts/suggestions.
>>> >>>> Regards,
>>> >>>> Amogh
>>>
>>

Re: Request for review of PR [Beam-8564]

[Amogh Tiwari]

Hi Luke,

I agree with your thoughts and observations. But, airlift:aircompressor is
the only implementation of LZO in pure java. That straight away solves #5.
The other implementations that I found either have licensing issues (since
LZO natively uses GNU GPL licence) or are implemented using .c, .h and jni
(which again make them dependent on the OS). Please refer these:
twitter/hadoop-lzo
<https://github.com/twitter/hadoop-lzo/tree/master/src/main/java/com/hadoop/compression/lzo>
and
shevek/lzo-java
<https://github.com/shevek/lzo-java/tree/master/lzo-core/src/main/java/org/anarres/lzo>
.
These were the main reasons why we based this on airlift:aircompressor.

Thanks and Regards,
Amogh



On Tue, Dec 3, 2019 at 2:59 AM Luke Cwik  wrote:

> I took a look. My biggest concern is finding a good LZO implementation.
> Looking for one that preferably has:
> 1) Apache license
> 2) Has zero transitive dependencies
> 3) Is small
> 4) Is performant
> 5) Is native java or supports execution on the three main OSs (Windows,
> Linux, Mac)
>
> In your PR you suggested using io.airlift:aircompressor:0.16 which doesn't
> meet item #2 and its transitive dependency fails #3.
>
> On Mon, Dec 2, 2019 at 12:16 PM Amogh Tiwari  wrote:
>
>> Hi,
>> I have filed a PR for an extension that will enable Apache Beam to work
>> with LZO/LZOP compression. Please refer
>> <https://issues.apache.org/jira/browse/BEAM-8564>.
>> I would love it if someone can take this up and review it.
>> Please feel free to share your thoughts/suggestions.
>> Regards,
>> Amogh
>>
>

Request for review of PR [Beam-8564]

[Amogh Tiwari]

Hi,
I have filed a PR for an extension that will enable Apache Beam to work
with LZO/LZOP compression. Please refer
.
I would love it if someone can take this up and review it.
Please feel free to share your thoughts/suggestions.
Regards,
Amogh

Re: goVet and clickHouse tests failing

[Amogh Tiwari]

Adding +Sameer Abhyankar  for visibility.

On Fri, Nov 29, 2019 at 4:10 PM Amogh Tiwari  wrote:

> Hi Elliotte,
> Thanks for your reply.
> Tried out the steps that you suggested, still not able to resolve this
> issue.
> Should I use a particular version of go for the build?
> What I have noticed is that when I ./gradlew build, it automatically
> downloads go 1.12 and sets the go env for it, basically ignoring whatever
> version of go that I already have installed.
> Please follow the last comment of this
> https://issues.apache.org/jira/browse/BEAM-8564.
> Regards
> Amogh
>
>
> On Wed, Nov 27, 2019 at 5:46 PM Elliotte Rusty Harold 
> wrote:
>
>> I did get through this one, and made the classic mistake of not
>> immediately committing the steps I took to writing. I believe it
>> involved some combination of setting go paths in environment
>> variables. I seem to have added this to the end of my .profile:
>>
>> export GOROOT=/usr/local/go
>> export GOPATH=$HOME/go/packages
>> export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
>>
>> According to my history, I also ran
>>
>> go get github.com/linkedin/goavro
>>
>> I really need to dedicate some time to cleaning up the build
>> documentation. See https://issues.apache.org/jira/browse/BEAM-8798
>>
>> Right now we have overlapping and sometimes contradictory docs in four
>> different places:
>>
>> README.md
>> CONTRIBUTING.md
>> https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ
>> https://beam.apache.org/contribute/
>>
>> We should probably pick one as the source of truth and rewrite the
>> other three to simply point to it. I propose putting all checkout,
>> build, test, commit, and push instructions in CONTRIBUTING.md in the
>> repo. What do folks think?
>>
>> On Wed, Nov 27, 2019 at 3:51 AM Amogh Tiwari  wrote:
>> >
>> > Hi Elliotte,
>> > I am facing a similar goVet issue. It would be great if you can guide
>> me through the solution. Please let me know the steps that you followed.
>> > Regards,
>> > Amogh
>> >
>> > On Thu, Nov 21, 2019 at 6:06 PM Elliotte Rusty Harold <
>> elh...@ibiblio.org> wrote:
>> >>
>> >> Tentatively, the goVet issue does seem to have been an issue with my
>> >> Go install I have now cleaned up. The clickhouse issue remains, as do
>> >> several others I'm working through.
>> >>
>> >> I've filed https://issues.apache.org/jira/browse/BEAM-8798 to
>> >> consolidate and update the instructions for getting to a working
>> >> build.
>> >>
>> >> On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold
>> >>  wrote:
>> >> >
>> >> > I'm slowly working my way through getting the tests to run and pass.
>> >> > We have a lot of work to do on the contributing docs to explain how
>> to
>> >> > setup and run the build. There's clearly a lot of knowledge in
>> >> > developers' heads and workstations that hasn't yet made it into the
>> >> > docs.
>> >> >
>> >> > The latest is a problem finding "github.com/linkedin/goavro" when I
>> >> > run goVet. I'm not a go person. Is this something that requires an
>> >> > extra install? If so, how is it installed? Or is this some error in
>> >> > the build.gradles? Or perhaps my go config is borked and gradle is
>> >> > looking in the wrong directory?
>> >> >
>> >> > > Task :sdks:go:examples:resolveBuildDependencies
>> >> > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go
>> >> > .gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2
>> :
>> >> > cannot find package "github.com/linkedin/goavro" in any of:
>> >> >
>>  /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/
>> github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro
>> >> > (vendor tree)
>> >> > /home/elharo/.gradle/go/binary/1.12/go/src/
>> github.com/linkedin/goavro
>> >> > (from $GOROOT)
>> >> >
>>  /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/
>> github.com/linkedin/goavro
>> >> > (from $GOPATH)
>> >> >
>> >> > > Task :sdks:go:examples:goVet FAILED
>> >> >
>> >> > I'm also seeing failures in ClickHouseIOTest:
>> >> >
>> >> > > Task :sdks:java:io:clickhouse:test
>> >> >
>> >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod
>> FAILED
>> >> > java.lang.IllegalStateException
>> >> >
>> >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod
>> FAILED
>> >> > java.lang.NullPointerException
>> >> >
>> >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod
>> FAILED
>> >> > java.lang.IllegalStateException
>> >> >
>> >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod
>> FAILED
>> >> > java.lang.NullPointerException
>> >> >
>> >> > --
>> >> > Elliotte Rusty Harold
>> >> > elh...@ibiblio.org
>> >>
>> >>
>> >>
>> >> --
>> >> Elliotte Rusty Harold
>> >> elh...@ibiblio.org
>>
>>
>>
>> --
>> Elliotte Rusty Harold
>> elh...@ibiblio.org
>>
>

Re: goVet and clickHouse tests failing

[Amogh Tiwari]

Hi Elliotte,
Thanks for your reply.
Tried out the steps that you suggested, still not able to resolve this
issue.
Should I use a particular version of go for the build?
What I have noticed is that when I ./gradlew build, it automatically
downloads go 1.12 and sets the go env for it, basically ignoring whatever
version of go that I already have installed.
Please follow the last comment of this
https://issues.apache.org/jira/browse/BEAM-8564.
Regards
Amogh


On Wed, Nov 27, 2019 at 5:46 PM Elliotte Rusty Harold 
wrote:

> I did get through this one, and made the classic mistake of not
> immediately committing the steps I took to writing. I believe it
> involved some combination of setting go paths in environment
> variables. I seem to have added this to the end of my .profile:
>
> export GOROOT=/usr/local/go
> export GOPATH=$HOME/go/packages
> export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
>
> According to my history, I also ran
>
> go get github.com/linkedin/goavro
>
> I really need to dedicate some time to cleaning up the build
> documentation. See https://issues.apache.org/jira/browse/BEAM-8798
>
> Right now we have overlapping and sometimes contradictory docs in four
> different places:
>
> README.md
> CONTRIBUTING.md
> https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ
> https://beam.apache.org/contribute/
>
> We should probably pick one as the source of truth and rewrite the
> other three to simply point to it. I propose putting all checkout,
> build, test, commit, and push instructions in CONTRIBUTING.md in the
> repo. What do folks think?
>
> On Wed, Nov 27, 2019 at 3:51 AM Amogh Tiwari  wrote:
> >
> > Hi Elliotte,
> > I am facing a similar goVet issue. It would be great if you can guide me
> through the solution. Please let me know the steps that you followed.
> > Regards,
> > Amogh
> >
> > On Thu, Nov 21, 2019 at 6:06 PM Elliotte Rusty Harold <
> elh...@ibiblio.org> wrote:
> >>
> >> Tentatively, the goVet issue does seem to have been an issue with my
> >> Go install I have now cleaned up. The clickhouse issue remains, as do
> >> several others I'm working through.
> >>
> >> I've filed https://issues.apache.org/jira/browse/BEAM-8798 to
> >> consolidate and update the instructions for getting to a working
> >> build.
> >>
> >> On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold
> >>  wrote:
> >> >
> >> > I'm slowly working my way through getting the tests to run and pass.
> >> > We have a lot of work to do on the contributing docs to explain how to
> >> > setup and run the build. There's clearly a lot of knowledge in
> >> > developers' heads and workstations that hasn't yet made it into the
> >> > docs.
> >> >
> >> > The latest is a problem finding "github.com/linkedin/goavro" when I
> >> > run goVet. I'm not a go person. Is this something that requires an
> >> > extra install? If so, how is it installed? Or is this some error in
> >> > the build.gradles? Or perhaps my go config is borked and gradle is
> >> > looking in the wrong directory?
> >> >
> >> > > Task :sdks:go:examples:resolveBuildDependencies
> >> > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go
> >> > .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2
> :
> >> > cannot find package "github.com/linkedin/goavro" in any of:
> >> >
>  /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro
> >> > (vendor tree)
> >> > /home/elharo/.gradle/go/binary/1.12/go/src/
> github.com/linkedin/goavro
> >> > (from $GOROOT)
> >> >
>  /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/
> github.com/linkedin/goavro
> >> > (from $GOPATH)
> >> >
> >> > > Task :sdks:go:examples:goVet FAILED
> >> >
> >> > I'm also seeing failures in ClickHouseIOTest:
> >> >
> >> > > Task :sdks:java:io:clickhouse:test
> >> >
> >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod
> FAILED
> >> > java.lang.IllegalStateException
> >> >
> >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod
> FAILED
> >> > java.lang.NullPointerException
> >> >
> >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod
> FAILED
> >> > java.lang.IllegalStateException
> >> >
> >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod
> FAILED
> >> > java.lang.NullPointerException
> >> >
> >> > --
> >> > Elliotte Rusty Harold
> >> > elh...@ibiblio.org
> >>
> >>
> >>
> >> --
> >> Elliotte Rusty Harold
> >> elh...@ibiblio.org
>
>
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org
>

Re: goVet and clickHouse tests failing

[Amogh Tiwari]

Hi Elliotte,
I am facing a similar goVet issue. It would be great if you can guide me
through the solution. Please let me know the steps that you followed.
Regards,
Amogh

On Thu, Nov 21, 2019 at 6:06 PM Elliotte Rusty Harold 
wrote:

> Tentatively, the goVet issue does seem to have been an issue with my
> Go install I have now cleaned up. The clickhouse issue remains, as do
> several others I'm working through.
>
> I've filed https://issues.apache.org/jira/browse/BEAM-8798 to
> consolidate and update the instructions for getting to a working
> build.
>
> On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold
>  wrote:
> >
> > I'm slowly working my way through getting the tests to run and pass.
> > We have a lot of work to do on the contributing docs to explain how to
> > setup and run the build. There's clearly a lot of knowledge in
> > developers' heads and workstations that hasn't yet made it into the
> > docs.
> >
> > The latest is a problem finding "github.com/linkedin/goavro" when I
> > run goVet. I'm not a go person. Is this something that requires an
> > extra install? If so, how is it installed? Or is this some error in
> > the build.gradles? Or perhaps my go config is borked and gradle is
> > looking in the wrong directory?
> >
> > > Task :sdks:go:examples:resolveBuildDependencies
> > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go
> > .gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2
> :
> > cannot find package "github.com/linkedin/goavro" in any of:
> > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/
> github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro
> > (vendor tree)
> > /home/elharo/.gradle/go/binary/1.12/go/src/
> github.com/linkedin/goavro
> > (from $GOROOT)
> > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/
> github.com/linkedin/goavro
> > (from $GOPATH)
> >
> > > Task :sdks:go:examples:goVet FAILED
> >
> > I'm also seeing failures in ClickHouseIOTest:
> >
> > > Task :sdks:java:io:clickhouse:test
> >
> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED
> > java.lang.IllegalStateException
> >
> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED
> > java.lang.NullPointerException
> >
> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED
> > java.lang.IllegalStateException
> >
> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED
> > java.lang.NullPointerException
> >
> > --
> > Elliotte Rusty Harold
> > elh...@ibiblio.org
>
>
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org
>

Re: Permission to contribute on LZO compression enablement for Beam Java SDK

[Amogh Tiwari]

Thanks Luke :)

On Fri, Nov 8, 2019 at 1:01 AM Luke Cwik  wrote:

> Welcome, I have added you as a contributor and assigned the ticket to you.
>
> On Thu, Nov 7, 2019 at 4:21 AM Amogh Tiwari  wrote:
>
>> Hi,
>>
>> I would like to contribute on enabling Apache Beam's java SDK to work
>> with LZO compression. Please add me as a contributor so that I can work on
>> this.
>> I've also raised a ticket
>> <https://issues.apache.org/jira/browse/BEAM-8564> for the same.
>>
>> Thanks and best regards,
>> Amogh Tiwari
>>
>

Permission to contribute on LZO compression enablement for Beam Java SDK

[Amogh Tiwari]

Hi,

I would like to contribute on enabling Apache Beam's java SDK to work with
LZO compression. Please add me as a contributor so that I can work on this.
I've also raised a ticket <https://issues.apache.org/jira/browse/BEAM-8564>
for the same.

Thanks and best regards,
Amogh Tiwari