Re: [Announcement] New Website for Beam Summits

Great initiative, thanks for creating this! Btw, are any plans to add there information about previous Beam-related events, especially London Beam summit last year? > On 20 Mar 2019, at 19:30, David Morávek wrote: > > This is great! Thanks for all of the hard work you're putting into this. >

Re: [PROPOSAL] commit granularity in master

Etienne, thanks for bringing this topic. I think it was already discussed several times before and we have finally came to what we have in the current Committer guide “Granularity of changes" [1]. Personally, I completely agree with these 4 rules presented there. The main concern is that all co

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark! > On 25 Mar 2019, at 18:36, Mark Liu wrote: > > Thank you all! It's a great pleasure to work on Beam! > > Mark > > On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu > wrote: > Congratulations, Mark! > > On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri

Re: New contributor

Welcome, Guobao! Great to have you on board! Alexey > On 26 Mar 2019, at 11:49, Guobao Li wrote: > > Hi all, > > I am Guobao Li from Talend. I am new to Apache Beam. Currently I am working > on the implementation of CouchbaseIO and hope to contribute in more areas in > the future. > > I hav

Re: Debugging :beam-sdks-java-io-hadoop-input-format:test

Hi Mikhail, We had a flaky “HIFIOWithEmbeddedCassandraTest” a while ago and it was caused by issue with launching of embedded Cassandra cluster. Then it was fixed by Etienne Chauchot's PR [1] Though, I don’t see any similar error messages in your Jenkins job log, so, I’m not sure it’s the same

Re: Quieten javadoc generation

+1 to suppress such warnings globally. IMO, usually, meaningful Javadoc description is quite enough to understand what this method does. > On 1 Apr 2019, at 18:21, Kenneth Knowles wrote: > > Personally, I would like to suppress the warnings globally. I think requiring > javadoc everywhere is a

Re: kafka 0.9 support

> How are multiple versions of Kafka supported? Are they all in one client, or > is there a case for forks like ElasticSearchIO? They are supported in one client but we have additional “ConsumerSpEL” adapter which unifies interface difference among different Kafka client versions (mostly to sup

Re: Debugging :beam-sdks-java-io-hadoop-input-format:test

gt; https://builds.apache.org/job/beam_PreCommit_Java_Commit/5011/consoleFull > <https://builds.apache.org/job/beam_PreCommit_Java_Commit/5011/consoleFull> > On Thu, Mar 28, 2019 at 8:55 AM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > Hi Mikhail, > >

Re: kafka 0.9 support

...@gmail.com>> wrote: > >>>>> > >>>>> > >>>>> We're still using Kafka 0.10 a lot, similar as 0.9 IMO. To expand > >>>>> multiple versions in KafkaIO is quite complex now, and it confuses > >>

Re: kafka 0.9 support

Do we all agree that we can drop Kafka 0.9/0.10 support after next LTS release (is it already known which one will be the next?) or we need to vote for this? > On 4 Apr 2019, at 16:49, Alexey Romanenko wrote: > > +1 for that too. > >> On 4 Apr 2019, at 01:53, Raghu An

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

> since early 2018 > 100+ pull requests Wow, this is impressive! Great job, congrats! > On 11 Apr 2019, at 15:08, Maximilian Michels wrote: > > Great work! Congrats. > > On 11.04.19 13:41, Robert Bradshaw wrote: >> Congratulations! >> On Thu, Apr 11, 2019 at 12:29 PM Michael Luckey wrote: >>>

Re: Wait on JdbcIO write completion

ly(Wait.on(firstWriteResults)) .apply(JdbcIO.write().withDataSourceConfiguration(CONF_DB_2)); [1] https://issues.apache.org/jira/browse/BEAM-6732 <https://issues.apache.org/jira/browse/BEAM-6732> > On 22 Feb 2019, at 16:52, Alexey Romanenko wrote: > > I have created new Jira issue

Re: CassandraIO breakage

How it fails? No issue for me with local build against master. > On 17 Apr 2019, at 21:48, Reuven Lax wrote: > > Did something break with CassandraIO? It no longer seems to compile.

Re: [PROPOSAL] Prepare for LTS bugfix release 2.7.1

Thanks for working on this, Kenn. Perhaps, I missed this but has it been already discussed/decided what will be the next LTS release? > On 26 Apr 2019, at 08:02, Kenneth Knowles wrote: > > Since it is all trivially reversible if there is some other feeling about > this thread, I have gone ahe

Re: :beam-sdks-java-io-hadoop-input-format:test is extremely flaky

Despite the fact that after fixing an issue with ports allocation (thanks to Etienne!) for embedded Cassandra cluster (it’s used in hadoop-input-format and this was the main cause of flakiness) it's got much better, I’m 100% pro to remove this module since it’s already been deprecated for severa

Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

Congrats! > On 2 May 2019, at 10:06, Gleb Kanterov wrote: > > Congratulations! Well deserved! > > On Thu, May 2, 2019 at 10:00 AM Ismaël Mejía > wrote: > Congrats everyone ! > > On Thu, May 2, 2019 at 9:14 AM Robert Bradshaw > wrote: > C

Re: kafka client interoperability

Thank you for reporting this. Seems like it’s a bug there (since ProducerRecord from kafka-clients:0.10.2.1 doesn’t support headers), so I created a Jira for that: https://issues.apache.org/jira/browse/BEAM-7217 Unfortunately, I can’t reproduce

Re: kafka client interoperability

Oops, I see that Richard already created a Jira about that, so I close mine as a duplicate. > On 3 May 2019, at 15:58, Alexey Romanenko wrote: > > Thank you for reporting this. > > Seems like it’s a bug there (since ProducerRecord from kafka-clients:0.10.2.1 > doesn’t su

Re: :beam-sdks-java-io-hadoop-input-format:test is extremely flaky

emove it on this release, this is a maintenance pain for no real >> reason. >> >> On Mon, Apr 29, 2019 at 3:06 PM Alexey Romanenko >> wrote: >>> >>> Despite the fact that after fixing an issue with ports allocation (thanks >>> to Etienne!) for embedde

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congratulations, Udi! Thanks for your work! > On 4 May 2019, at 01:24, Heejong Lee wrote: > > Congratulations! > > On Fri, May 3, 2019 at 3:53 PM Reza Rokni > wrote: > Congratulations ! > > From: Reuven Lax mailto:re...@google.com>> > Date: Sat, 4 May 2019, 06:42 > To:

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congratulations, Pablo! > On 16 May 2019, at 20:38, Rui Wang wrote: > > Congrats! Congrats! Congrats! > > -Rui > > On Thu, May 16, 2019 at 9:45 AM Udi Meiri > wrote: > Congrats Pablo! > > On Thu, May 16, 2019 at 9:27 AM Thomas Weise > wrote:

Re: [BEAM-6673] pre-commit checks failing due to test failure in unrelated Docker module

Charith, Yes, it was caused by changing a scope of "kafka-clients" dependency. Now it should work fine (I run “JavaPortabilityApi” test on your PR and it’s green), thanks to Maximilian Michels for quick fix. Sorry for inconvenience. > On 20 May 2019, at 16:24, Maximilian Michels wrote: > > I

Re: RedisIO refactoring

Varun, thank you for starting this topic. I agree that it would make sense to do some refactoring and introduce something like “RedisRecord” which will be parameterised and contain all additional metadata if needed. In the same time, we can keep current public API based on KV, but move to using

Re: RedisIO refactoring

/AdvancedUsage#transactions <https://github.com/xetorthio/jedis/wiki/AdvancedUsage#transactions> > > > On Tue, May 21, 2019 at 4:55 PM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > Varun, thank you for starting this topic. > > I agree that it woul

Re: Measuring element sizes in benchmarks

On 28 May 2019, at 17:31, Łukasz Gajowy wrote: > > I'm not quite following what these sizes are needed for--aren't the > benchmarks already tuned to be specific, known sizes? > > Maybe I wasn't clear enough. Such metric is useful mostly in IO tests - > different IOs generate records of differen

Re: [DISCUSS] Cookbooks for users with knowledge in other frameworks

+1, sounds good for me! A while ago, I also was thinking in similar way that, potentially, some kind of FAQ/Cookbook would be helpful for users since I see from time to time the similar questions on user@, Slack and SO. Later, we could extend it wider and add the examples/solutions for different

PTransform.expand() guarantees

Hello, I tried to find an answer in documentation for the questions below but I haven’t managed to do that. Actually, there are 3 related questions: Does Beam guarantee where (at “driver” or at "worker” of backend system) "PTransform.expand()” of provided transform will be called? Does Beam g

Re: pipeline status tracing

I see that similar questions happen quite often in the last time, so, probably, it would make sense to add this “Wait.on()”-pattern to corresponding website documentation page [1]. [1] https://beam.apache.org/documentation/patterns/file-processing-patterns/

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

Congrats, Mikhail! > On 21 Jun 2019, at 18:01, Anton Kedin wrote: > > Congrats! > > On Fri, Jun 21, 2019 at 3:55 AM Reza Rokni > wrote: > Congratulations! > > On Fri, 21 Jun 2019, 12:37 Robert Burke, > wrote: > Congrats > > On Fri, Jun 21,

Re: PTransform.expand() guarantees

Thank you for answers, Lukasz. > On 21 Jun 2019, at 18:15, Lukasz Cwik wrote: > Does Beam guarantee where (at “driver” or at "worker” of backend system) > "PTransform.expand()” of provided transform will be called? > No. There are usecases where the driver is run in the "cloud" such as > tem

Re: PTransform.expand() guarantees

On 21 Jun 2019, at 19:12, Lukasz Cwik wrote: > This question was triggered by one issue in KinesisIO [1] where we did quite > expensive check that stream exists in Write.expand() (calling backend Kinesis > for that) and, according to issue reporter, it was called several times, > though I alway

Re: WebSocket/Https connector for Apache Beam (Java)?

Probably, this blog, which Romain Manni-Bucau wrote a while ago, could be helpful: https://rmannibucau.metawerx.net/post/apache-beam-websocket-output It’s not in Beam upstream but you can use it as your own transform. > On 2 J

Re: [ANNOUNCE] New committer: Robert Burke

Congratulations, Robert! > On 17 Jul 2019, at 14:49, Tim Robertson wrote: > > Congratulations Robert! > > On Wed, Jul 17, 2019 at 2:47 PM Gleb Kanterov > wrote: > Congratulations, Robert! > > On Wed, Jul 17, 2019 at 1:50 PM Robert Bradshaw

beam_PreCommit_Java_Commit is broken

Hi all, Looks like https://builds.apache.org/job/beam_PreCommit_Java_Commit/ is broken. It fails with: FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/

Re: beam_PreCommit_Java_Commit is broken

thon people take a look on this? Thank you. > On 5 Aug 2019, at 18:04, Alexey Romanenko wrote: > > Hi all, > > Looks like https://builds.apache.org/job/beam_PreCommit_Java_Commit/ > <https://builds.apache.org/job/beam_PreCommit_Java_Commit/> is broken. It > fails wi

Re: beam_PreCommit_Java_Commit is broken

Thank you for tracking and fixing this! > On 5 Aug 2019, at 18:26, Rui Wang wrote: > > It's being tracked by https://issues.apache.org/jira/browse/BEAM-7892 > <https://issues.apache.org/jira/browse/BEAM-7892>. > > > -Rui > > On Mon, A

Re: Waht would be the best place for performance tests documentation?

I agree with Cyrus that Confluence page should a good place for that since, seems, it will be very dev oriented documentation. > On 7 Aug 2019, at 16:31, Cyrus Maden wrote: > > Hi Łukasz, > > This sounds perfect for the confluence, since we already have the testing > guide >

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

Congrats, well deserved! > On 27 Aug 2019, at 11:25, Jan Lukavský wrote: > > Congrats Valentyn! > On 8/26/19 11:43 PM, Rui Wang wrote: >> Congratulations! >> >> >> -Rui >> >> On Mon, Aug 26, 2019 at 2:36 PM Hannah Jiang > > wrote: >> Congratulations Valentyn, we

Re: [discuss] How we support our users on Slack / Mailing list / StackOverflow

Pablo, thank you for raising this question. I can’t say for Python, but as a someone, who tries to keep an eye on Java SDK related questions on ML/Slack/SO for a while, I’d say that Slack is not very effective for this. There are several reasons for that: - People tend to expect a quick feedbac

Re: Pointers on Contributing to Structured Streaming Spark Runner

Hi Xinyu, Great to hear that you wish to contribute into new Spark runner! We used to have the sync meetings about all Spark runners in general every two weeks, so feel free to let know us if you want to participate too. Also, as one of the contributors for Structural Streaming Spark runner (ye

Re: Pointers on Contributing to Structured Streaming Spark Runner

I don’t see any updates on my calendar. Does it work for others? > On 19 Sep 2019, at 17:16, Ismaël Mejía wrote: > > 25/09 looks ok. I just updated the meeting invitation to the new > date.I will prepare a mini agenda in the shared minute document in the > meantime. > I cannot see the old invit

Re: [ANNOUNCE] New committer: Alan Myrvold

Congratulations! Well deserved! > On 30 Sep 2019, at 19:12, Udi Meiri wrote: > > Congrats Alan! > > On Mon, Sep 30, 2019 at 11:12 AM Alan Myrvold > wrote: > Thanks!! Looking forward to making more impact to Apache Beam > > On Mon, Sep 30, 2019 at 10:56 AM Mikhail G

Re: [Proposal] Apache Beam Event's Calendar

+1 for Calendar, excellent idea! If would be great if it will be on the website as well. Regards, Alexey > On 27 Jan 2018, at 08:10, Ron Gonzalez wrote: > > +1 to the calendar... > > Thanks, > Ron > > On Friday, January 26, 2018, 2:27:34 PM PST, Griselda Cuevas > wrote: > > > Ok, so give

Re: [PROPOSAL] Switch from Guava futures vs Java 8 futures

+1, sounds great! Regards, Alexey > On 2 Feb 2018, at 07:14, Thomas Weise wrote: > > +1 > > > On Thu, Feb 1, 2018 at 9:07 PM, Jean-Baptiste Onofré > wrote: > +1 > > Regards > JB > > On 02/01/2018 07:54 PM, Kenneth Knowles wrote: > > Hi all, > > > > Luke, Thomas, a

Re: [DISCUSS] What to do about widespread KinesisIO breakage

Kenn, JB, Thank you for raising this question. I was working mostly on writing support for KinesisIO (PR is waiting to be merged) but I’ll take a look on flaky read tests as well. Regards, Alexey > On 6 Feb 2018, at 06:27, Jean-Baptiste Onofré wrote: > > Yes, I don't consider this as blocker

Re: Code reviews in Beam

Reuven, thank you for bringing this topic. As a new contributor to Beam codebase I raise two my hands for such guideline document and I'd propose to add it as a new guide into section “Other Guides” on web site documentation. For sure, there are already several very helpful and detailed guides

Jenkins job beam_PreCommit_Python_MavenInstall” still fails

Hi all, Jenkins job “beam_PreCommit_Python_MavenInstall” has been constantly failing for last 3 days. Last successful build (#3052) has been produced at Feb 16: https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/lastSuccessfulBuild/

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

ste Onofré wrote: > > Yes it seems to be related to the change on the Python direct runner. > > Did you try to revert the change to see if it fixes the build ? > > Thanks > Regards > JB > Le 20 févr. 2018, à 16:13, Alexey Romanenko <mailto:aromanenko@gmail.com&

Re: [DISCUSS] What to do about widespread KinesisIO breakage

org/jira/browse/BEAM-3317>), I can’t reproduce constantly. Even running it thousands times I’ve seen it only once. So, it’s still in progress. WBR, Alexey > On 6 Feb 2018, at 11:40, Alexey Romanenko wrote: > > Kenn, JB, > > Thank you for raising this question. > I was w

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

lt;https://issues.apache.org/jira/browse/BEAM-3720> > > > > On Tue, Feb 20, 2018 at 4:43 PM, Jean-Baptiste Onofré <mailto:j...@nanthrax.net>> wrote: > > I would wait a feedback from the original author of the commit. > > > > Regards > > JB >

Re: Publishing release artifacts to custom artifactory

tory" controls the addition of another maven repo to read > dependencies from, it doesn't impact project publishing and shouldn't be > needed. > > On Fri, Aug 24, 2018 at 5:28 AM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > Maybe my answer

Re: Beam Schemas: current status

Thanks Reuven for updating community with this, great work! One small question about IO integration. What kind of integration this is supposed to be? Are there any IOs that already benefit from Schemas support? > On 31 Aug 2018, at 16:46, Reuven Lax wrote: > > > > On Fri, Aug 31, 2018 at 2:2

Re: Should we mention TF Transform in Beam site?

+1 Perhaps, it would be also helpful to add a small “get started” example to attract user attention. > On 5 Sep 2018, at 02:16, Ahmet Altay wrote: > > +1 > > It would be nice to showcase other tools built on top of Beam. > > On Mon, Aug 20, 2018 at 1:02 PM, Ankur Goenka

Re: [PROPOSAL] Prepare Beam 2.7.0 release

Charles, Does it happen when you run this task separately or as a part of the whole project build? I run it independently and don’t see any issues on my side. Also, I don’t remember that it failed on Jenkins recently. So, probably this segfault is not related to exactly this task. > On 5 Sep 20

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Welcome to Beam community, Dat Tran! Thank you for your work! > On 6 Sep 2018, at 10:10, Etienne Chauchot wrote: > > Hi all , > > Just to let you know that Elasticsearch IO now supports version 6 in addition > to version 2 and 5 now that this PR is merged: > https://github.com/apache/beam/pul

[DISCUSS] Unification of Hadoop related IO modules

Hello everyone, I’d like to discuss the following topic (see below) with community since the optimal solution is not clear for me. There is Java IO module, called “hadoop-input-format”, which allows to use MapReduce InputFormat implementations to read data from different sources (for example,

Re: [DISCUSS] Unification of Hadoop related IO modules

tiste Onofré <mailto:j...@nanthrax.net>> wrote: > Hi, > > in order to limit the impact for the existing users on Beam 2.x series, > I would go for (1). > > Regards > JB > > On 06/09/2018 17:24, Alexey Romanenko wrote: > > Hello everyone, > > > > I’d

Re: [DISCUSS] Unification of Hadoop related IO modules

day we can read from Hadoop FS but can't write to > Hadoop FS using Beam HDFS API ? > > Regards > Dharmendra > > On Thu, Sep 6, 2018 at 8:54 PM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > Hello everyone, > > I’d like to discuss the followi

Re: [PROPOSAL] Test performance of basic Apache Beam operations

I agree that we can benefit from having two types of performance tests (low and high level) that could complement each other. Can we detect a regression (if any) automatically and send a report about that? Sorry if we already do that for Nexmark. > On 11 Sep 2018, at 11:29, Etienne Chauchot wro

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

+1 Great feature that should help with complicated error cases. > On 11 Sep 2018, at 03:39, Henning Rohde wrote: > > +1 Nice proposal. It will help eradicate some of the inflexibility and > frustrations with Jenkins. > > On Wed, Sep 5, 2018 at 2:30 PM Yifan Zou > w

Re: [DISCUSS] Unification of Hadoop related IO modules

op file system abstraction itself has > implementation for multiple other systems and is not limited to HDFS. > > On Tue, Sep 11, 2018 at 8:47 AM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > Dharmendra, > For now, you can’t write with Hadoop MapReduce OutputFormat

Re: [DISCUSS] Unification of Hadoop related IO modules

gt; wrote: > On Wed, Sep 12, 2018 at 5:27 PM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > Thank you everybody for your feedback! > > I think we can conclude that the most popular option, according to discussion > above, is number 3. Not sure if we need to do

Re: [VOTE] Release 2.7.0, release candidate #1

Perhaps it could help, but I run simple WordCount (built with Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine for me. > On 14 Sep 2018, at 06:56, Romain Manni-Bucau wrote: > > Hi Charles, > > I didn't get enough time to check deeply but it is clearly a dependency issue > and

Re: Compatibility Matrix vs Runners in the code base

I agree that we should keep all runners in Matrix table but it should be clear which runners are part of Beam code base, which are not. In case of non-Beam code repo runners we need to provide a link to its documentation. MapReduce runner is another case since it’s not yet production ready and

Re: Agenda for the Beam Summit London 2018

I wish to all to have a great summit in London! Unfortunately, I can’t be there, so, if the talks will be recorded and available on-line later then it would be amazing! > On 30 Sep 2018, at 12:37, Pascal Gula wrote: > > Hi, > Do any of you plan to meet on Sunday evening for dinner? > See you

Re: Modular IO presentation at Apachecon

Interesting talk, I’ve had a chance to see this in person. Many thanks to Ismael and Eugene! > On 27 Sep 2018, at 21:34, Eugene Kirpichov wrote: > > Thanks Ismael and everyone else! Unfortunately I do not believe that this > session was recorded on video :( > Juan - yes, this is some of the im

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

+1 to what Anton said, I’m fully agree with this. Looking on the current commit history we can notice that there are many commits like “fix”, “add/edit/remove”, “clean up”, etc, that, seems, were introduced after the review process iterations. I think this makes commit history too much verbose

Re: Are there plans for removing joda-time from the beam java SDK?

Jeff, thank you for raising this topic. + 1 to final move to java.time in Beam 3.0 I didn’t find a Jira about that, so I created a new one: https://issues.apache.org/jira/browse/BEAM-5530 > On 27 Sep 2018, at 15:37, Jeff Klukas wrote: > > I ag

Re: [PROPOSAL] Prepare Beam 2.8.0 release

Regarding LTS release - I agree that we need to have clear view what kind of support will be provided for such releases. Despite of the concerns mentioned before, I have another one about API labeled as “@Experimental". I think there are most of IOs, SQL, PCollection with Schema, etc, labeled w

Re: Beam website sources migrated to apache/beam

This is great news! Many thanks to all who contributed to this! I believe that as the process of website update will be clearer and simpler, as our documentation will be up-to-date. > On 6 Oct 2018, at 06:38, Jean-Baptiste Onofré wrote: > > Awesome, > > thanks for the move Scott ! > > Regard

Re: [DISCUSS] Gradle for the build ?

I agree with all previous points related to Gradle-related problems. Among all others, I’d highlight IDE integration issue since it can have real negative impact on attracting new contributors and, finally, community grow. The more obstacles there are to start code contribution, the more chances

Re: [FYI] Paper of Building Beam Runner for IBM Streams

Really great paper, thank you for sharing! > On 10 Sep 2018, at 18:38, Tim wrote: > > Thanks for sharing Manu - interesting paper indeed. > > Tim > >> On 10 Sep 2018, at 16:02, Maximilian Michels wrote: >> >> Excellent write-up. Thank you! >> >>> On 09.09.18 20:43, Jean-Baptiste Onofré wrot

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

+1 > On 16 Oct 2018, at 00:02, Chamikara Jayalath wrote: > > +1 for new lists. > > Thanks, > Cham > > On Mon, Oct 15, 2018 at 12:09 PM Ismaël Mejía > wrote: > +1 > On Mon, Oct 15, 2018 at 8:14 PM Mikhail Gryzykhin > wrote: > > > > +1 also s

Re: [ANNOUNCE] New committers & PMC members, Summer 2018 edition

ributions to Beam, Thomas is PMC chair for >>>>Apache Apex. >>>> >>>>New committers: >>>> >>>> - Charles Chen, longtime contributor to Python SDK, Python direct >>>>runner, state & timers >>>> - Ł

Re: New Edit button on beam.apache.org pages

This is really cool feature! With a tab “Preview changes” it makes documentation updating much more easier to do. Thanks a lot to Alan and Scott! > On 25 Oct 2018, at 09:48, Robert Bradshaw wrote: > > Very cool! Thanks! > On Thu, Oct 25, 2018 at 9:38 AM Connell O'Callaghan > wrote: >> >> A

Re: :beam-sdks-java-io-hadoop-input-format:test task issues

Alex, could you check and update (if necessary) the package "libjffi-jni” on your system? Other potential workaround could be to try to run it with “ -Dcom.datastax.driver.USE_NATIVE_CLOCK=false" > On 31 Oct 2018, at 23:40, Kenneth Knowles wrote: > > If I am reading it right, the segfault is

Re: JB's back

This is great to have you back, JB! Take care! > On 22 Nov 2018, at 11:36, Łukasz Gajowy wrote: > > Welcome back JB! > > czw., 22 lis 2018 o 07:58 Robert Burke > napisał(a): > Welcome back! > > On Wed, Nov 21, 2018, 11:46 AM Rui Wang > wr

Re: Edit access to the Apache Beam Confluence Wiki?

And for me, please! Thanks! > On 22 Nov 2018, at 14:28, Łukasz Gajowy wrote: > > Could anyone give me the access too? I'd like to update some Nexmark related > instructions. > > Thanks! > > wt., 30 paź 2018 o 00:05 Alan Myrvold > napisał(a): > Thanks! > > > On M

Re: Stand at FOSDEM 2019

I’m going to visit FOSDEM this year as well and will be glad to help as much as I can. > On 30 Nov 2018, at 13:00, Maximilian Michels wrote: > > Thank you for all your reactions. Looks like we will have a great presence at > FOSDEM :) > > @Matthias: Yes, I'm planning to go. > @Wout: Locals

Issue with publishing maven artefacts locally

Hi all, I used to publish maven artefacts into local repository using this kind of command for example: ./gradlew -Ppublishing --no-parallel -PdistMgmtSnapshotsUrl=file:///path/to/.m2/repository/ -p sdks/java/io/kafka/ publish It worked fine till today. Seems like (according to "git bisect”)

Re: [Call for items] ❄️ December Beam Newsletter

I've just added several things that were finished. Thanks for driving this! > On 7 Dec 2018, at 23:39, Rose Nguyen wrote: > > Hi folks: > > Time for the last newsletter of the year! > > Add to [1] the highlights from November to now (or planned events and talks) > that you want to share by 1

Re: Issue with publishing maven artefacts locally

hing in PR#7197 [1] > >>> > >>> +Garrett Jones, who authored the change. Garrett can you help investigate? > >>> > >>> I went to check to see if we have any existing Jenkins jobs that would've > >>> caught this break. It seems the

Re: [ANNOUNCE] Apache Beam 2.9.0 released!

Congrats! Thank you to all contributors and to Chamikara for driving this release! > On 14 Dec 2018, at 10:32, Tim wrote: > > Thank you for running the release Chamikara. > > Tim, > Sent from my iPhone > > On 14 Dec 2018, at 10:30, Matt Casters > wrote: > >> Gre

Re: [DISCUSS] Unification of Hadoop related IO modules

ache.org/jira/browse/BEAM-6246> > On 13 Sep 2018, at 20:26, Alexey Romanenko wrote: > > Robert, Chamikara, > Yes, I agree that we need to give enough time for that. I’m fine to wait > until 3.0 > >> On 12 Sep 2018, at 19:27, Chamikara Jayalath > <mailto:chami

Re: Beam snapshots broken

Thank you for fixing this, Scott. I also can confirm that it works for me now. > On 2 Jan 2019, at 22:36, Ismaël Mejía wrote: > > Thanks Scott, I just tested and the errors I found are fixed and > everything seems to work now. Snapshots are being correctly updated > too. > > On Wed, Jan 2, 2019

Re: Dev contact for KafkaIO

ogle and Dataflow team. I had a great time > and learnt lot from working on both Apache Beam and Dataflow. > > My new job at a startup is not directly related to Apache Beam and I will not > be able to spend a lot of time on support or development of KafkaIO. > > Alexey Rom

Re: Joining an Unbounded Source with a Bounded Source

In case of joining bounded and unbounded sources with CoGroupByKey, I can guess that all data from bounded source (MySQL in this case) just comes into only one (first) window because all elements have the same timestamp (minimum timestamp or “-infinity" used for bounded sources, afaik). So, in o

Re: Connection leaks with PostgreSQL instance

Kenn, I’m not sure that we have a connection leak in JdbcIO since new connection is being obtained from an instance of javax.sql.DataSource (created in @Setup) and which is org.apache.commons.dbcp2.BasicDataSource by default. BasicDataSource uses connection pool and closes all idle connections

Re: The full list of proposals / prototype documents

nt to this page in the same time as it was created and finalised after all discussions. It should help us to keep this list consistent and always up-to-date. Thank you in advance! On 11 Jul 2018, at 17:03, Alexey Romanenko wrote: > > Thank you for this link, Etienne. > I agree that it d

Re: The full list of proposals / prototype documents

I’m sorry but I forgot to mention that the whole list could be found here: https://beam.apache.org/contribute/design-documents/ <https://beam.apache.org/contribute/design-documents/> > On 18 Jan 2019, at 16:49, Alexey Romanenko wrote: > > FYI: I updated the list of design docum

Re: The full list of proposals / prototype documents

well, I tried 2) proposals require access approval. Should > that be the case? > > _/ > _/ Alex Van Boxel > > > On Fri, Jan 18, 2019 at 4:51 PM Alexey Romanenko <mailto:aromanenko@gmail.com>> wrote: > I’m sorry but I forgot to mention that the whole lis

Re: [ANNOUNCE] New committer announcement: Gleb Kanterov

Congrats to Gleb and welcome on board! > On 25 Jan 2019, at 09:22, Tim Robertson wrote: > > Welcome Gleb and congratulations! > > On Fri, Jan 25, 2019 at 8:06 AM Kenneth Knowles > wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new commi

Re: BEAM-6324 / #7340: "I've pretty much given up on the PR being merged. I use my own fork for my projects"

For sure, it’s a pity that this PR has not been addressed for a long time (I guess, we probably have other ones like this) but, as I can see from this PR history, review has not been requested explicitly by author (and this is one of the our recommendations for code contribution [1]). What are

Re: The full list of proposals / prototype documents

Wpw5oGAJDv0X7y9OSnPnrZWJk>] > Reject Unsupported Windowing Strategies in JOIN [doc > <https://docs.google.com/document/d/1V-ZgKVTwHdNSGlQWncWIzcf_Rw2oLKZFSkU43scLff4>] > the ones that I had access to, from the past I don't know anymore of course. > > _/ > _/ Ale

Re: [ANNOUNCE] New PMC member: Etienne Chauchot

Great job! Congrats, Etienne! > On 28 Jan 2019, at 07:18, Ahmet Altay wrote: > > Congratulations Etienne! > > On Sun, Jan 27, 2019 at 7:15 PM Reza Ardeshir Rokni > wrote: > Congratulations Etienne! > > On Sat, 26 Jan 2019 at 14:16, Ismaël Mejía

Re: New contributor: Michał Walenia

Welcome on board, Michał! > On 31 Jan 2019, at 10:17, Reza Ardeshir Rokni wrote: > > Welcome! > > On Thu, 31 Jan 2019 at 15:48, Michał Walenia > wrote: > HI all, > thanks for a warm welcome :) > > Michał > >> Wiadomość napisana przez Ahmet Altay >

Re: Another another new contributor! :)

Welcome, Kyle! Great to have more people on SparkRunner side! Btw, feel free to join “beam-spark” channel on Slack. Alexey > On 7 Feb 2019, at 08:44, Reza Ardeshir Rokni wrote: > > Welcome! > > On Tue, 5 Feb 2019 at 23:34, Kenneth Knowles > wrote: > Welcome Kyle! >

Re: [VOTE] Release 2.10.0, release candidate #3

+1 I tested briefly new HadoopFormatIO (read/write), deprecated HadoopInputFormatIO (read) and new KafkaIO feature (write into multiple topics) > On 7 Feb 2019, at 12:01, Maximilian Michels wrote: > > +1 (binding) > > On 06.02.19 23:47, Reuven Lax wrote: >> +1 (binding) >> On Wed, Feb 6, 201

Re: pipeline steps

Talking about KafkaIO, it’s already possible to have this since "apply(KafkaIO.read())" returns "PCollection>” where KafkaRecord contains message metadata (topic, partition, etc). Though, it works _only_ if “withoutMetadata()” was not used before - in this case it will return simple KV. In th

Re: Empty projects defined in settings.xml

Hmm, I’m confused since this is me, who is author of this commit [1], but these 2 project definitions are absolutely not related to HadoopFormatIO.Write Do we need to drop it? [1] https://github.com/apache/beam/commit/757b71e749ab8a9f0a08e3669596ce69920acbac

  1   2   3   4   5   6   >