Re: :beam-sdks-java-io-hadoop-input-format:test task issues

2018-10-31 Thread Kenneth Knowles
If I am reading it right, the segfault is not in the Java compiler, but in the method https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Native.html#isGettimeofdayAvailable-- called when HIFIO is testing with embedded Cassandra. Kenn On Wed, Oct 31, 2018 at 2:59 PM Alex Amato

Re: :beam-sdks-java-io-hadoop-input-format:test task issues

2018-10-31 Thread Alex Amato
Personally, I don't care too much if there is a test which fails often. I would just like to track these tests and have some test target I can run locally which will run most of the tests. So if this can't be easily fixed, If we could have a separate gradle target where we blacklist a few problem

Re: :beam-sdks-java-io-hadoop-input-format:test task issues

2018-10-31 Thread Ruoyun Huang
+1 to have inputs regarding this failure Alex raised. javaPreCommit never worked on my local machine in past a few weeks. This hadoop build target has been the main issue. On Tue, Oct 30, 2018 at 5:31 PM Alex Amato wrote: > Hello, > > I keep encountering issues with the precommit process, and

Re: Flink 1.6 Support

2018-10-31 Thread Jins George
Thanks Max. That would help a lot. Manually creating the build with 1.6 will work for us in short term with development, but to go to production it has to be an official release ( due to some regulations at my company) Thanks, Jins On 10/31/18 3:15 AM, Maximilian Michels wrote: > Hi Jins, > >

Re: Follow up ideas, to simplify creating MonitoringInfos.

2018-10-31 Thread Lukasz Cwik
I see and don't know how to help you beyond what your already suggesting. >From what I remember, maps were added as syntactic sugar of lists of key value pairs. On Tue, Oct 30, 2018 at 5:37 PM Alex Amato wrote: > I am not sure on the correct syntax to populate the instances of my >

Re: Accessing keyed state in portable timer callbacks

2018-10-31 Thread Lukasz Cwik
I filed https://issues.apache.org/jira/browse/BEAM-5930. On Wed, Oct 31, 2018 at 10:22 AM Lukasz Cwik wrote: > That looks like a bug in the FnApiDoFnRunner.java > > The FnApiStateAccessor is given a callback to get the current element and > it is not handling the case where the current element

Re: Data Preprocessing in Beam

2018-10-31 Thread Kenneth Knowles
The word "extension" doesn't really mean anything in the case of Beam. It is just a library. You can use the build set up of other libraries as examples. Kenn On Wed, Oct 31, 2018 at 10:23 AM Alejandro wrote: > Hello, > > I am going to get familiarized on how to write a Beam extension then, >

Re: Data Preprocessing in Beam

2018-10-31 Thread Alejandro
Hello, I am going to get familiarized on how to write a Beam extension then, although right now I am a little busy searching for a new job :-/. I hope in a few weeks (Lets hope it doesn't take much longer to find a job) I can get hands on it this and contribute with this preprocessing extension

Re: Accessing keyed state in portable timer callbacks

2018-10-31 Thread Lukasz Cwik
That looks like a bug in the FnApiDoFnRunner.java The FnApiStateAccessor is given a callback to get the current element and it is not handling the case where the current element is a timer. callback:

Accessing keyed state in portable timer callbacks

2018-10-31 Thread Maximilian Michels
Hi, I have a question regarding user state during timer callback in the FnApiDoFnRunner (Java SDK Harness). I've started implementing Timers for the portable Flink Runner. I can register a timer via the timer output collection and fire the timer via the timer input of the SDK Harness. But

Re: PCollectionViews$SimplePCollectionView.hashCode allocates memory

2018-10-31 Thread Vojtech Janota
Ok, will do both. Thanks, Vojta On Wed, Oct 31, 2018 at 2:32 PM Ismaël Mejía wrote: > Vojta you are right, your implementation seems like a good improvement. > Can you please create a JIRA and eventually if you are interested do a > PR to contribute a fix for it. > > Regards, > Ismaël > On

Re: PCollectionViews$SimplePCollectionView.hashCode allocates memory

2018-10-31 Thread Ismaël Mejía
Vojta you are right, your implementation seems like a good improvement. Can you please create a JIRA and eventually if you are interested do a PR to contribute a fix for it. Regards, Ismaël On Wed, Oct 31, 2018 at 2:18 PM Vojtech Janota wrote: > > Hi, > > I'm currently profiling memory

PCollectionViews$SimplePCollectionView.hashCode allocates memory

2018-10-31 Thread Vojtech Janota
Hi, I'm currently profiling memory consumption of our Beam pipeline and have noticed that org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.hashCode() makes noticeable heap allocations. The implementation is: return Objects.hash(tag); That itself translates to: return

Re: Data Preprocessing in Beam

2018-10-31 Thread Ismaël Mejía
Hello, I mentored Arnaud to contribute the sketching extension into Beam and from a quick look at Alex paper + implementation, I think this should be an independent extension. Sketching is a collection of transforms that rely on probabilistic data structures to give approximate results and

Re: bigquery issue

2018-10-31 Thread Ismaël Mejía
Hello, If you think it is a bug (or issue) you can report it at Apache's JIRA https://issues.apache.org/jira/projects/BEAM/issues If it is more of a use related question probably it is better to do it in the user@ mailing list. Notice that reporting issues is a way of contributing to Beam, for

Re: Flink 1.6 Support

2018-10-31 Thread Maximilian Michels
Hi Jins, As Thomas mentioned, the Flink Runner has already been prepared for Flink 1.6, you just have to change the Flink version in the Gradle build file. Of course this is not convenient because you can't fetch this version via Maven Central. So we're planning to release both versions:

bigquery issue

2018-10-31 Thread Chaim Turkel
Hi, I have an issue with the bigquery sdk code, where is the correct group to send them? chaim -- Loans are funded by FinWise Bank, a Utah-chartered bank located in Sandy, Utah, member FDIC, Equal Opportunity Lender. Merchant Cash Advances are made by Behalf. For more information on ECOA,