Re: PCollection to PCollection Conversion

I would love to see a lean core and abundant Transforms at the same time. Maybe we can look at what Confluent does for kafka-connect. They have official extensions support for JDBC, HDFS and ElasticSearch under https://github.com/confluentinc. They put them along

Re: [DISCUSS] Change "RunnableOnService" To A More Intuitive Name

I think it's important to tease apart what why we're trying to mark tests. Generally, nearly all tests should run on all runners. However, there are some exceptions, namely. 1) Some runners don't support all features (especially at the start). 2) Some tests are incompatible with distributed

Re: [DISCUSS] Change "RunnableOnService" To A More Intuitive Name

Hi Mark, Generally speaking, I agree. As RunnableOnService extends NeedsRunner, @TestsWithRunner or @RunOnRunner sound clearer. Regards JB On 11/09/2016 09:00 PM, Mark Liu wrote: Hi all, I'm working on building RunnableOnService in Python SDK. After having discussions with folks,

[DISCUSS] Change "RunnableOnService" To A More Intuitive Name

Hi all, I'm working on building RunnableOnService in Python SDK. After having discussions with folks, "RunnableOnService" looks like not a very intuitive name for those unit tests that require runners and build lightweight pipelines to test specific components. Especially, they don't have to run

Re: SBT/ivy dependency issues

Hi Abbass, Seeing the output from `sbt dependency-tree` from the sbt-dependency-graph plugin [1] might help. (caveat: I did not try this out; I don't know the state of maintenance) Kenn [1] https://github.com/jrudolph/sbt-dependency-graph On Wed, Nov 9, 2016 at 6:33 AM, Jean-Baptiste Onofré

Re: PCollection to PCollection Conversion

On this point from Amit and Ismaël, I agree: we could benefit from a place for miscellaneous non-core helper transformations. We have sdks/java/extensions but it is organized as separate artifacts. I think that is fine, considering the nature of Join and SortValues. But for simpler transforms,

Re: [PROPOSAL] Merge apex-runner to master branch

Hi Thomas, Very good point about establishing more clear definitions of the roles mentioned in the guidelines. Let's discuss in a separate thread. Kenn On Tue, Nov 8, 2016 at 1:03 PM, Thomas Weise wrote: > Thanks for the support. It may be helpful to describe the roles of >

Re: SBT/ivy dependency issues

Hi Abbass, As discussed together, it could be related to some changes we did in the Maven profiles and build. Let me investigate. I keep you posted. Thanks ! Regards JB On 11/09/2016 03:03 PM, amarouni wrote: Hi guys, I'm facing a weird issue with a Scala project (using SBT/ivy) that

Re: PCollection to PCollection Conversion

Related to spark-package, we also have Apache Bahir to host connectors/transforms for Spark and Flink. IMHO, right now, Beam should host this, not sure if it makes sense directly in the core. It reminds me the "Integration" DSL we discussed in the technical vision document. Regards JB On

Re: PCollection to PCollection Conversion

I think Jesse has a very good point on one hand, while Luke's and Kenneth's worries about committing users to specific implementations is in place. The Spark community has a 3rd party repository for useful libraries that for various reasons are not a part of the Apache Spark project: