Common model for runners

2018-03-20 Thread Ron Gonzalez 
Hi,  When I build a data flow using the Beam SDK, can someone point me to the 
code that represents the underlying representation of the beam model itself?  
Is there an API that lets me retrieve the underlying protobuf-based graph for 
the data flow? Perhaps some pointers to what code in the runner retrieves this 
model in order to execute it in the specific engine?
Thanks,Ron

Re: [Proposal] Apache Beam Event's Calendar

2018-01-26 Thread Ron Gonzalez 
 +1 to the calendar...
Thanks,Ron
On Friday, January 26, 2018, 2:27:34 PM PST, Griselda Cuevas 
 wrote:  
 
 Ok, so given that the proposal seem to have traction I'll proceed as follows: 
   
   - I'll give all PMC members owner rights to the calendar, so they can edit 
and add events. 
   - I'll share the calendar with the dev@ and user@ lists
   - I'll investigate how to add the calendar to the Website
Proposed dynamic to add events to the calendar:   
   - Anyone can announce an event in the mailing list 
   - Anyone with editing rights in the calendar can add a new event
[For people with Calendar editing rights only] How to add an event to the 
calendar:   
   - I'll suggest we use [tags] to classify events in this calendar:
[Deadline] -- Use this tag for reminders on deadlines for CFPs, Registrations, 
etc.   
[USA] -- Use this tag for events in the USA   
[Europe] -- Use this tag for events in Europe   
[APAC] -- Use this tag for events in APAC
   - We should add speakers, participants or content directly in the calendar 
space for details
Cheers, G
On 25 January 2018 at 01:35, Ismaël Mejía  wrote:

+1

I think it makes sense to separate it the calendar in two, one for the
CFPs more interesting for dev@ and one for confirmed events where
there will be presentations on Beam that concerns more the users
(user@). It also probably makes sense to include this one in the
website too.




On Thu, Jan 25, 2018 at 9:46 AM, Etienne Chauchot  wrote:
> +1, great initiative!
>
>
> Le 25/01/2018 à 01:05, Griselda Cuevas a écrit :
>
> Hi Beam Community,
>
> I've created this public calendar to curate events that the Apache Beam
> community will attend or is organizing, as well as industry events we should
> keep in the radar. With this calendar, I want to:
>
> Give visibility to the community on what conferences we want to participate
> in
> What events is our community organizing
>
>
> My proposal: Share the calendar with the entire dev@ & users@ mailing list
> and give access as admins to the few folks I know are organizing events to
> help curate our events.
>
> If we get enough votes I'll share the calendar publicly and I'll give access
> to the other folks.
>
> Thanks,
> G
>
>

Re: Eclipse support

2018-01-26 Thread Ron Gonzalez 
 Cool thanks Daniel.
Thanks,Ron
On Friday, January 26, 2018, 1:31:17 PM PST, Daniel Kulp <dk...@apache.org> 
wrote:  
 
 
We do have some documentation at:
https://beam.apache.org/contribute/eclipse/

But it doesn’t really work anymore.  I submitted a PR:
https://github.com/apache/beam/pull/4502
to use as a starting point to try and get things running in Eclipse again, but 
it’s not completely there yet.    That said, I’m using Eclipse Oxygen, not 
Neon, which brings a completely new set of challenges.  :(

The PR will get the poms importing via m2e without error, apt running for the 
value types, javacc and it related things running for those things, and pretty 
much get all the generated code and the source directories and everything 
visible in eclipse so the compiler has a chance.      However, I still have 20 
errors related to generics that I may need help resolving.  I’ll try and look 
into it more early next week, I hope.  (Also need to copy 
~/.m2/repository/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.jar
 into your /Applications/Eclipse.app/Contents/Eclipse/plugins or equivalent 
directory)

The eclipse-jdt profile uses an older version of Tyco and the jdt it pulls in 
(0.26).  The PR does update Tyco to 1.0.0 which is closer to the newer eclipse 
compiler, but still not exact.  


The other thing that generates a TON of errors in Eclipse is Checkstyle.  The 
latest Checkstyle plugin in Eclipse is based on Checkstyle 8.7.  Beam uses 
Checkstyle 6.19.  The configurations are not compatible.  Longer term, we would 
need to update Beam to 8.x, but that may be a huge PR of mostly just formatting 
changes.  Checkstyle has become EXTREMELY picky about a few things.  It took 
quite a bit to get CXF updated to 8.x, touching most of the files.  I’m not 
sure how much of an impact it would have on Beam.  I’ll probably investigate 
that next week as well, but that may require some discussions around what the 
code SHOULD look like in the areas where the old checkstyle didn’t care but the 
new one does.  We’ll see.  


Dan






> On Jan 23, 2018, at 5:14 AM, Ismaël Mejía <ieme...@gmail.com> wrote:
> 
> Hello again,
> 
> The current compiler configuration in maven should be ok. So you can
> test that the compilation works from outside of eclipse by running.
> 
>    mvn clean compile test-compile -Peclipse-jdt
> 
> The fix is more about making the compiler happy about some use of
> Java's generics and some other small code changes of this kind, for
> reference you can take a look at the fixes Dan Kulp did for this
> previously:
> 
> https://github.com/apache/beam/commit/a0ae04bef40149cdf54d0ab50909f18a444f3023
> https://github.com/apache/beam/commit/957c545eaa33c861b561418b1c7dadf4c31f92f3
> 
> Regards,
> Ismaël
> 
> On Tue, Jan 23, 2018 at 2:56 AM, zlgonzalez <zlgonza...@yahoo.com> wrote:
>> Thanks Ismael. I'll try and take a stab at it when I've read more about the
>> eclipse compiler.
>> 
>> In the meantime, do you have any pointers? Is it just about finding the
>> right Eclipse JDT compiler options?
>> 
>> Thanks,
>> Ron
>> 
>> 
>> 
>> Sent via the Samsung Galaxy S7 active, an AT 4G LTE smartphone
>> 
>>  Original message 
>> From: Ismaël Mejía <ieme...@gmail.com>
>> Date: 1/22/18 1:29 AM (GMT-08:00)
>> To: dev@beam.apache.org
>> Subject: Re: Eclipse support
>> 
>> Hello,
>> 
>> Thanks for bringing this info, I tried to compile with the eclipse
>> compiler and I can confirm that it does not wok, Eclipse's JDT is more
>> annoying about generics so it could be related to this.
>> 
>> Filled https://issues.apache.org/jira/browse/BEAM-3508 to track it.
>> Feel free to contribute a fix if you feel like it.
>> 
>> Ismaël
>> 
>> 
>> On Sat, Jan 20, 2018 at 10:20 PM, Ron Gonzalez <zlgonza...@yahoo.com> wrote:
>>> Hello again,
>>>  Sorry to keep asking about this, but I can't seem to get Eclipse working
>>> for this project.
>>> 
>>>  I did mvn eclipse:clean eclipse:eclipse command and I've reduced the
>>> problem now down to the Java generics issues related to autovalue types. I
>>> had to run mvn generate-sources generate-test-sources in each of the
>>> sub-modules to make a lot of the errors in Eclipse go away since after
>>> running mvn -DskipTests clean install, the target/generated-sources and
>>> target/generated-test-sources are empty for some reason.
>>> 
>>>  Interestingly enough, if I run mvn -Peclipse-jdt -DskipTests clean
>>> install
>>> from the command line, I am able to reproduce the same errors that I see
>>> in
>>> my Eclipse ins

Re: Eclipse support

2018-01-17 Thread Ron Gonzalez 
 Yes I forgot to mention that. I have done that.So the main problems are the 
following:
1. The generated code (like RunnerApi.java) seems to be deleted by the time the 
build is done. By just manully compiling beam/model and refreshing the project, 
I was able to resolve it and reduce the errors from 1200+ to 113, which 
correspond to the remaining issues #2 and #3 below.
2.  Not sure if this is a JDK8 issue. Right now, importing sets it to 
JDK7.Description Resource Path Location TypeBound mismatch: The type 
Combine.PerKey is not a valid substitute for the bounded parameter 
> of the type 
AppliedPTransform<InputT,OutputT,TransformT> CombineTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 82 Java Problem
3. This is particularly strange, but I'm still trying to sort it out. 
Description Resource Path Location TypeColumn cannot be resolved to a type 
AutoValue_Column.java 
/beam-sdks-java-extensions-sql/target/generated-sources/annotations/org/apache/beam/sdk/extensions/sql/meta
 line 8 Java Problem
Thanks,Ron

On Wednesday, January 17, 2018, 10:34:48 AM PST, Ted Yu 
<yuzhih...@gmail.com> wrote:  
 
 Have you tried running 'mvn eclipse:eclipse' and importing from the root of 
workspace ?
On Wed, Jan 17, 2018 at 10:32 AM, Ron Gonzalez <zlgonza...@yahoo.com> wrote:

Hi,  I've been trying this for a couple of days now, but I can't seem to get a 
clean Eclipse import.  I refreshed to latest master, got a clean mvn 
-DskipTests clean install, ran through the Eclipse setup steps for m2e-apt 
installation.  I'm getting errors like below. Do you have any tips to get this 
going?
Thanks,Ron
Description Resource Path Location TypeACCUMULATING cannot be resolved to a 
variable WindowingStrategyTranslation. java /beam-runners-core- 
construction-java/src/main/ java/org/apache/beam/runners/ core/construction 
line 56 Java ProblemAFTER_ALL cannot be resolved to a variable 
TriggerStateMachines.java /beam-runners-core-java/src/ 
main/java/org/apache/beam/ runners/core/triggers line 34 Java ProblemAFTER_ALL 
cannot be resolved to a variable TriggerTranslation.java /beam-runners-core- 
construction-java/src/main/ java/org/apache/beam/runners/ core/construction 
line 241 Java ProblemAFTER_ANY cannot be resolved to a variable 
TriggerStateMachines.java /beam-runners-core-java/src/ 
main/java/org/apache/beam/ runners/core/triggers line 37 Java ProblemAFTER_ANY 
cannot be resolved to a variable TriggerTranslation.java /beam-runners-core- 
construction-java/src/main/ java/org/apache/beam/runners/ core/construction 
line 243 Java ProblemAFTER_EACH cannot be resolved to a variable 
TriggerStateMachines.java /beam-runners-core-java/src/ 
main/java/org/apache/beam/ runners/core/triggers line 59 Java ProblemAFTER_EACH 
cannot be resolved to a variable TriggerTranslation.java /beam-runners-core- 
construction-java/src/main/ java/org/apache/beam/runners/ core/construction 
line 245 Java ProblemAFTER_END_OF_WINDOW cannot be resolved to a variable 
TriggerStateMachines.java /beam-runners-core-java/src/ 
main/java/org/apache/beam/ runners/core/triggers line 40 Java 
ProblemAFTER_END_OF_WINDOW cannot be resolved to a variable 
TriggerTranslation.java /beam-runners-core- construction-java/src/main/ 
java/org/apache/beam/runners/ core/construction line 248 Java 
ProblemAFTER_PROCESSING_TIME cannot be resolved to a variable 
TriggerStateMachines.java /beam-runners-core-java/src/ 
main/java/org/apache/beam/ runners/core/triggers line 62 Java 
ProblemAFTER_PROCESSING_TIME cannot be resolved to a variable 
TriggerTranslation.java /beam-runners-core- construction-java/src/main/ 
java/org/apache/beam/runners/ core/construction line 276 Java 
ProblemAFTER_SYNCHRONIZED_PROCESSING_ TIME cannot be resolved to a variable 
TriggerStateMachines.java /beam-runners-core-java/src/ 
main/java/org/apache/beam/ runners/core/triggers line 45 Java 
ProblemAFTER_SYNCHRONIZED_PROCESSING_ TIME cannot be resolved to a variable 
TriggerTranslation.java /beam-runners-core- construction-java/src/main/ 
java/org/apache/beam/runners/ core/construction line 302 Java ProblemALIGN_TO 
cannot be resolved to a variable TriggerStateMachines.java 
/beam-runners-core-java/src/ main/java/org/apache/beam/ runners/core/triggers 
line 94 Java ProblemALIGN_TO cannot be resolved to a variable 
TriggerTranslation.java /beam-runners-core- construction-java/src/main/ 
java/org/apache/beam/runners/ core/construction line 281 Java ProblemALWAYS 
cannot be resolved to a variable TriggerStateMachines.java 
/beam-runners-core-java/src/ main/java/org/apache/beam/ runners/core/triggers 
line 51 Java ProblemALWAYS cannot be resolved to a variable 
TriggerTranslation.java /beam-runners-core- construction-java/src/main/ 
java/org/apache/beam/runners/ core/construction line 304 Java 
ProblemApiServiceDescriptor cannot be resolved GrpcFnServer.java 
/beam-runners-java-fn- execution/src/main/java/org

Eclipse support

2018-01-17 Thread Ron Gonzalez 
Hi,  I've been trying this for a couple of days now, but I can't seem to get a 
clean Eclipse import.  I refreshed to latest master, got a clean mvn 
-DskipTests clean install, ran through the Eclipse setup steps for m2e-apt 
installation.  I'm getting errors like below. Do you have any tips to get this 
going?
Thanks,Ron
Description Resource Path Location TypeACCUMULATING cannot be resolved to a 
variable WindowingStrategyTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 56 Java ProblemAFTER_ALL cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 34 Java ProblemAFTER_ALL cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 241 Java ProblemAFTER_ANY cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 37 Java ProblemAFTER_ANY cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 243 Java ProblemAFTER_EACH cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 59 Java ProblemAFTER_EACH cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 245 Java ProblemAFTER_END_OF_WINDOW cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 40 Java ProblemAFTER_END_OF_WINDOW cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 248 Java ProblemAFTER_PROCESSING_TIME cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 62 Java ProblemAFTER_PROCESSING_TIME cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 276 Java ProblemAFTER_SYNCHRONIZED_PROCESSING_TIME cannot be resolved to 
a variable TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 45 Java ProblemAFTER_SYNCHRONIZED_PROCESSING_TIME cannot be resolved to a 
variable TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 302 Java ProblemALIGN_TO cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 94 Java ProblemALIGN_TO cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 281 Java ProblemALWAYS cannot be resolved to a variable 
TriggerStateMachines.java 
/beam-runners-core-java/src/main/java/org/apache/beam/runners/core/triggers 
line 51 Java ProblemALWAYS cannot be resolved to a variable 
TriggerTranslation.java 
/beam-runners-core-construction-java/src/main/java/org/apache/beam/runners/core/construction
 line 304 Java ProblemApiServiceDescriptor cannot be resolved GrpcFnServer.java 
/beam-runners-java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution
 line 37 Java Problem


Thanks,Ron

Re: Strata Conference this March 6-8

2018-01-16 Thread Ron Gonzalez 
 Works for me...
On Tuesday, January 16, 2018, 5:45:33 PM PST, Holden Karau 
<hol...@pigscanfly.ca> wrote:  
 
 How would folks feel about during the afternoon break (3:20-4:20) on the 
Wednesday (same day as Eugene's talk)? We could do the Philz which is a bit of 
a walk but gets us away from the big crowd and also lets folks not attending 
the conference but in the area join us.
On Tue, Jan 16, 2018 at 5:29 PM, Ron Gonzalez <zlgonza...@yahoo.com> wrote:

 Cool, let me know if you guys finally schedule it. I will definitely try to 
make it to Eugene's talk but having an informal BoF in the area would be nice...
Thanks,Ron
On Tuesday, January 16, 2018, 5:06:53 PM PST, Boris Lublinsky 
<boris.lublin...@lightbend.com > wrote:  
 
 All for it
Boris Lublinsky
FDP Architect
boris.lublin...@lightbend.com
https://www.lightbend.com/

On Jan 16, 2018, at 7:01 PM, Ted Yu <yuzhih...@gmail.com> wrote:
+1 to BoF
On Tue, Jan 16, 2018 at 5:00 PM, Dmitry Demeshchuk <dmi...@postmates.com> wrote:

Probably won't be attending the conference, but totally down for a BoF.
On Tue, Jan 16, 2018 at 4:58 PM, Holden Karau <hol...@pigscanfly.ca> wrote:

Do interested folks have any timing constraints around a BoF?
On Tue, Jan 16, 2018 at 4:30 PM, Jesse Anderson <je...@bigdatainstitute.io> 
wrote:

+1 to BoF. I don't know if any Beam talks will be on the schedule.

> We could do an informal BoF at the Philz nearby or similar?




-- 
Twitter: https://twitter.com/h oldenkarau




-- 
Best regards,Dmitry Demeshchuk.



  



-- 
Twitter: https://twitter.com/holdenkarau

Some interesting use case

2018-01-16 Thread Ron Gonzalez 
Hi,  I was wondering if anyone has encountered or used Beam in the following 
manner:   1. During machine learning training, use Beam to create the event 
table. The flow may consist of some joins, aggregations, row-based 
transformations, etc...  2. Once the model is created, deploy the model to some 
scoring service via PMML (or some other scoring service).  3. Enable the SAME 
transformations used in #1 by using a separate engine but thereby guaranteeing 
that it will transform the data identically as the engine used in #1.
  I think this is a pretty interesting use case where Beam is used to guarantee 
portability across engines and deployment (batch to true streaming, not 
micro-batch). What's not clear to me is with respect to how batch joins would 
translate during one-by-one scoring (probably lookups) or how aggregations 
given that some kind of history would need to be stored (and how much is kept 
is configurable too).
  Thoughts?
Thanks,Ron

Re: Strata Conference this March 6-8

2018-01-16 Thread Ron Gonzalez 
 Cool, let me know if you guys finally schedule it. I will definitely try to 
make it to Eugene's talk but having an informal BoF in the area would be nice...
Thanks,Ron
On Tuesday, January 16, 2018, 5:06:53 PM PST, Boris Lublinsky 
 wrote:  
 
 All for it
Boris Lublinsky
FDP Architect
boris.lublin...@lightbend.com
https://www.lightbend.com/

On Jan 16, 2018, at 7:01 PM, Ted Yu  wrote:
+1 to BoF
On Tue, Jan 16, 2018 at 5:00 PM, Dmitry Demeshchuk  wrote:

Probably won't be attending the conference, but totally down for a BoF.
On Tue, Jan 16, 2018 at 4:58 PM, Holden Karau  wrote:

Do interested folks have any timing constraints around a BoF?
On Tue, Jan 16, 2018 at 4:30 PM, Jesse Anderson  
wrote:

+1 to BoF. I don't know if any Beam talks will be on the schedule.

> We could do an informal BoF at the Philz nearby or similar?




-- 
Twitter: https://twitter.com/h oldenkarau




-- 
Best regards,Dmitry Demeshchuk.

Strata Conference this March 6-8

2018-01-16 Thread Ron Gonzalez 
Hi,  Will there be some talks or representation of Apache Beam at the coming 
Strata Conference this March 6-8?  Would be great to hear someone talk about 
how Beam's been used at their company as their core data integration platform.
Thanks,Ron