Re: [PROPOSAL] commit granularity in master

It is good to remind committers of their responsability on the 'cleanliness' of the merged code. Github sadly does not have an easy interface to do this and this should be done manually in many cases, sadly I have seen many committers just merging code with multiple 'fixup' style commits by clickin

Re: [PROPOSAL] commit granularity in master

0K issue so we will then have 3 chars less, probably it does not change much but still. On Fri, Mar 22, 2019 at 3:19 PM Robert Bradshaw wrote: > > On Fri, Mar 22, 2019 at 3:02 PM Ismaël Mejía wrote: > > > > It is good to remind committers of their responsability on the > >

Re: [spark runner dataset POC] workCount works !

gt;>>> wanted to remind that we have #beam-spark channel on Slack where we > >>>> discuss all ongoing questions. Feel free to join! > >>>> > >>>> Alexey > >>>> > >>>> > On 21 Mar 2019, at 15:51, Jean-Baptiste Onofré > >&g

Re: New contributor

Welcome Guobao! Nice that you are joining us. Looking forward for your contributions ! Take the time to read the contribution guide https://beam.apache.org/contribute/ and don't hesitate to ask any question you may have. On Tue, Mar 26, 2019 at 2:14 PM Alexey Romanenko wrote: > > Welcome, Guobao

SNAPSHOTS have not been updated since february

I was trying to validate a fix on the Spark runner and realized that Beam SNAPSHOTS have not been updated since February 24 ! https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.12.0-SNAPSHOT/ Can somebody please take a look at why this is not been u

Re: Deprecating Avro for fastavro on Python 3

Hello, The problem of switching implementations is the risk of losing interoperability, and this is more important than performance. Does fastavro have tests that guarantee that it is fully compatible with Avro’s Java version? (given that it is the de-facto implementation used everywhere). If per

Re: New contributor

Welcome Niklas! On Fri, Mar 29, 2019 at 3:54 PM Guobao Li wrote: > > Welcome! > > On Wed, Mar 27, 2019 at 11:12 PM Kenneth Knowles wrote: >> >> Welcome! >> >> On Wed, Mar 27, 2019 at 2:59 PM Mikhail Gryzykhin wrote: >>> >>> Welcome Niklas. >>> >>> This is another location with useful resources

Re: Contibutor permissions for Beam Jira tickets

You have now the Contributor role, and I assigned the ticket you asked for. Enjoy! Ismaël On Mon, Apr 1, 2019 at 12:35 PM Madhusudhan Reddy Vennapusa wrote: > > Hi, > > This is Madhu, I am interested to contribute to Apache Beam. > > Can someone please add me as contributor, I would like to assi

Re: [VOTE] Release 2.12.0, release candidate #1

-1 The release misses a cherry pick [1] that fixes an important issue in Cassandra, without this users won't be able to write to Cassandra. I know at least 3 users who are waiting for this release to have this fixed. [1] https://github.com/apache/beam/pull/8198/files On Wed, Apr 3, 2019 at 8:34

Re: kafka 0.9 support

We should focus on the main reason to remove the Kafka 0.9 support. I have the impression that this is mostly to ease the maintenance, but from the current status (and the removal PR [1]), it does not seem like it is a burden to continue supporting 0.9. In any case I am +1 to remove the support for

[DISCUSS] Backwards compatibility of @Experimental features

When we did the first stable release of Beam (2.0.0) we decided to annotate most of the Beam IOs as @Experimental because we were cautious about not getting the APIs right in the first try. This was a really good decision because we could do serious improvements and refactorings to them in the firs

Re: [VOTE] Release 2.12.0, release candidate #1

e some time so it is worth putting extra effort into pre-RC steps. > > Kenn > > [1] https://issues.apache.org/jira/projects/BEAM/versions/12344944 > [2] https://github.com/apache/beam/pulls?q=is:open+base:release-2.12.0 > > On Wed, Apr 3, 2019 at 12:39 PM Ismaël Mejía wrote: &g

Re: [DISCUSS] Backwards compatibility of @Experimental features

. However this >>>>>> forces us to think about each one. >>>>>> >>>>>> Downside - it might add more toil to the existing release process. >>>>>> >>>>>> Reuven >>>>>> >>>>>>

Re: [VOTE] Release 2.12.0, release candidate #3

-1 due to dependencies leaking in 'sdks/java/core'. For more details see BEAM-7042 [1] Michael promptly find a fix for the issue. (Thanks Michael!) Andrew you can find the cherry-pick PR here [2], and sorry for the extra work of another RC [1] https://issues.apache.org/jira/browse/BEAM-7042 [2] h

Re: Removing :beam-website:testWebsite from gradle build target

+1 to removing link validation for website changes. However it would be good to have a sort of weekly report on dead links or another alternative to be aware of them. On Tue, Apr 16, 2019 at 2:43 AM Kyle Weaver wrote: > I agree with Andrew that the external links checks are ultra-flaky and > sel

Re: SNAPSHOTS have not been updated since february

to build and publish >>>> artifacts and the postrelease_snapshot is to verify whether the snapshot >>>> works. >>>> >>>> On Tue, Mar 26, 2019 at 8:45 AM Ahmet Altay wrote: >>>>> >>>>> I believe this is related to >

Re: SNAPSHOTS have not been updated since february

And is there a way we can detect SNAPSHOTS not been published daily in the future? On Thu, Apr 18, 2019 at 6:37 PM Ismaël Mejía wrote: > > Any progress on this? > > On Wed, Mar 27, 2019 at 5:38 AM Daniel Oliveira > wrote: > > > > I made a bug for this specific issu

Re: SNAPSHOTS have not been updated since february

>> jenkin machine cannot publish artifacts into maven owing to some weird >>> permission issue. I think +Yifan Zou is working on it actively. >>> >>> On Thu, Apr 18, 2019 at 9:44 AM Ismaël Mejía wrote: >>>> >>>> And is there a wa

Re: CVE audit gradle plugin

I want to bring this subject back, any chance we can get this running in or main repo maybe in a weekly basis like we do for the dependency reports. It looks totallly worth. On Fri, Mar 1, 2019 at 2:05 AM Ahmet Altay wrote: > > Thank you, I agree this is very important. Does anyone know a similar

Re: Postcommit kiosk dashboard

Catching up on this one, nice dashboard ! Some jobs are misisng e.g. validatesRunner for both Spark and Flink. I suppose those are important if this may eventually replace the README as Thomas suggests. On Fri, Mar 15, 2019 at 2:18 AM Thomas Weise wrote: > > This is very nice! > > Perhaps it can

Re: Contributing Beam Kata (Java & Python)

This looks great, nice for bringing this to the project Henry! On Fri, Apr 19, 2019 at 10:53 AM hsuryawira...@google.com wrote: > > Thanks Altay. > I'll create it under "learning/" first as this is not exactly example. > Please do let me know if it's not the right place. > > On 2019/04/18 22:49:4

Re: [EXT] Re: [DOC] Portable Spark Runner

Thanks for sharing, the diagram really helps to understand. Please consider adding it to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Tue, Apr 16, 2019 at 12:00 AM Ankur Goenka wrote: > Thanks for sharing. > This looks great! > > On Mon, Apr 15, 2019 at

Re: [DISCUSS] Backwards compatibility of @Experimental features

a shorter > timelines assuming that we will set the user expectations right. > > On Fri, Apr 5, 2019 at 3:01 PM Ismaël Mejía wrote: > >> I agree 100% with Kenneth on the multiple advantages that the >> Experimental feature gave us. I also can count multiple places where this &

Re: Contributing Beam Kata (Java & Python)

+lars.fran...@gmail.com who is in the Apache training project and may be interested in this one or at least the JetBrains like approach. On Fri, Apr 19, 2019 at 12:01 PM Ismaël Mejía wrote: > > This looks great, nice for bringing this to the project Henry! > > On Fri, Apr 19, 2019

Re: SNAPSHOTS have not been updated since february

ck once our > nodes are fully setup. > > Thanks > > Yifan > > > On Fri, Apr 19, 2019 at 2:05 AM Ismaël Mejía wrote: >> >> Thanks everyone for the quick answer and thanks Yifan for taking care. >> >> On Thu, Apr 18, 2019 at 7:15 PM Yifan Zou wrote: >

Re: Contributing Beam Kata (Java & Python)

something along those lines could be interesting for hands-on type > things in training. > > On Fri, Apr 19, 2019 at 12:10 PM Ismaël Mejía wrote: >> >> +lars.fran...@gmail.com who is in the Apache training project and may >> be interested in this one or at least the Jet

Re: [VOTE] Release 2.12.0, release candidate #4

Etienne RC1 vote happened in 04/03 and there have not been any cherry picks on the spark runner afterwards so if there is a commit that degraded performance around 04/10 it is not part of the release we are voting, so please consider reverting your -1. However the issue you are reporting looks imp

Re: Hello from Hannah Jiang

Welcome to Beam Hannah ! On Fri, Apr 26, 2019 at 1:44 PM Maximilian Michels wrote: > > Awesome. Welcome Hannah! > > Cheers, > Max > > On 26.04.19 05:08, Yifan Zou wrote: > > Welcome！ > > > > On Thu, Apr 25, 2019 at 7:34 PM Connell O'Callaghan > > wrote: > > > > We

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

Ankur, do you have the equivalent link that I can use to subscribe to that calendar via google calendars? The link seems to work only to see the calendar in a webpage. Thanks. On Fri, Apr 26, 2019 at 1:42 PM Maximilian Michels wrote: > > Hi Ankur, > > Sounds good. This will ensure that we stay o

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

le calendar and prompts >> to add the Beam Calendar (at least in Chrome). Ismael, do you have a >> similar button in your case? >> >> [image: image.png] >> >> Regards, >> Anton >> >> >> On Fri, Apr 26, 2019 at 5:07 AM Ismaël Mejía wrote:

Re: :beam-sdks-java-io-hadoop-input-format:test is extremely flaky

+1 to remove it on this release, this is a maintenance pain for no real reason. On Mon, Apr 29, 2019 at 3:06 PM Alexey Romanenko wrote: > > Despite the fact that after fixing an issue with ports allocation (thanks to > Etienne!) for embedded Cassandra cluster (it’s used in hadoop-input-format >

Re: Updates on Beam Jenkins

Thanks Yifan for all your work. Sometimes the work on infrastructure is hidden, so it is great to acknowledge the importance of the improvements you and the others have done. On Mon, Apr 29, 2019 at 5:11 PM Lukasz Cwik wrote: > > Thanks Yifan for driving this. > > On Mon, Apr 29, 2019 at 8:01 AM

Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

Congrats everyone ! On Thu, May 2, 2019 at 9:14 AM Robert Bradshaw wrote: > Congratulation, and thanks for all the great contributions each one of you > has made to Beam! > > On Thu, May 2, 2019 at 5:51 AM Ruoyun Huang wrote: > >> Congratulations everyone! Well deserved! >> >> On Wed, May 1, 2

Re: [DISCUSS] Should File based IOs implement readAll() or just readFiles()

. On Wed, Feb 6, 2019 at 11:27 AM Jean-Baptiste Onofré wrote: > > +1 > > Thanks for that Ismaël. > > Regards > JB > > On 06/02/2019 11:24, Ismaël Mejía wrote: > > Since it seems we have consensus on deprecating both transforms I created > > >

Re: Contributing to Beam

Done, welcome! On Sat, May 4, 2019 at 3:43 AM Shehzaad Nakhoda wrote: > Hello > > I’m hoping to work with Rueven Lax (Google) on some enhancements and > existing issues. > > I would appreciate the ability to create and assign tickets to myself. > > My JIRA ID is shehzaadn. > > Thanks in advance!

Re: Access to JIRA

It should work now, enjoy! On Sat, May 4, 2019 at 7:58 PM Frederik Bode wrote: > > Hello everyone! > > I'm working on porting beam to python 3. Could I please get access > to JIRA? My JIRA username is frederik. > > Thanks, > Frederik > > [image: https://ml6.eu] >

Beam 2.14.0 SNAPSHOTS are broken

Hello, Just updated a project I have to verson 2.14.0-SNAPSHOT and found that the dependencies don't have the correct name, for example the beam-sdks-java-core pom [1] points to beam.model pipeline 2.14.0-SNAPSHOT compile But such dependency groupId / artifactId does not exist (or has

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congrats Pablo, well deserved, nece to see your work recognized! On Wed, May 15, 2019 at 9:59 AM Pei HE wrote: > > Congrats, Pablo! > > On Tue, May 14, 2019 at 11:41 PM Tanay Tummalapalli > wrote: > > > > Congratulations Pablo! > > > > On Wed, May 15, 2019, 12:08 Michael Luckey wrote: > >> > >>

Re: Beam 2.14.0 SNAPSHOTS are broken

BEAM-4046 artefact naming was adjusted. I ll have a look into that. >> >> Thanks for letting me know. >> >> Best, >> >> michel >> >> On Tue, May 14, 2019 at 5:29 PM Ismaël Mejía wrote: >>> >>> Hello, >>> >>> Just

Re: RedisIO refactoring

Hello Varun, This is an excellent idea because Redis already supports byte arrays as both keys and values. A more generic approach makes total sense. So worth a JIRA / PR. About the compatiblity concerns, RedisIO is tagged as @Experimental which means we can still evolve its API. Currently we are

Re: RedisIO refactoring

things, could we take that approach? It allows different elements to have > different methods and expirys. > > > > On Mon, May 20, 2019 at 11:30 PM Ismaël Mejía wrote: > >> Hello Varun, >> >> This is an excellent idea because Redis already supports byte arra

Re: Hazelcast Jet Runner

I saw that the runner was merged but I don’t get why the foler is called ‘runners/jet experimental’ and not simply ‘runners/jet’. Is it because the runner does not pass ValidatesRunner? Or because the contributors are few? I don’t really see any reason behind this suffix. And even if the status is

Re: Hazelcast Jet Runner

ote: > > Hi Ismaël! > > Quoting Kenn (from PR-8410): "We discussed on list that it would be better to > have new things always start as experimental in a way that clearly > distinguishes them from the core." > > Rgds > > On Thu, May 23, 2019 at 10:44 PM Ismaël Me

Re: Hazelcast Jet Runner

ly in the Gradle module name, >> but then there was some change >> ([BEAM-4046] decouple gradle project names and maven artifact ids - 4/2/19) >> which kind of ended up >> putting it in the directory name. Maybe I should have merged with that >> differently, but th

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1 On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > +1 > > On 27.05.19 14:04, Robert Bradshaw wrote: > > Sounds like everyone's onboard with the plan. Any chance we could > > publish these for the upcoming 2.13 release? > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: > >

[DISCUSS] Autoformat python code with Black

I stumbled by chance into Black [1] a python code auto formatter that is becoming the 'de-facto' auto-formatter for python, and wanted to bring to the ML Is there interest from the python people to get this into the build? The introduction of spotless for Java has been a good improvement and maybe

Re: RedisIO refactoring

at 6:44 PM Alexey Romanenko wrote: > On 21 May 2019, at 22:06, Ismaël Mejía wrote: > > > After a quick review of the code now I think I understand why it was > modeled as KV in the first place, the library that RedisIO > uses (Jedis) only supports 'mget' operation on St

Re: Hazelcast Jet Runner

on". I think releasing > under an "experimental" name is an improvement, making it available to users > to try out. But we probably should have discussed before doing something > different than all the other runners. > > Kenn > > [1] > https://lists.apache.org/th

Re: [DISCUSS] Autoformat python code with Black

t; This looks pretty good:) I know at least a couple people (myself >>> > included) who've been annoyed by having to take care of lint issues that >>> > maybe a code formatter could save us. >>> > Thanks for sharing Ismael. >>> > -P. >>> > >>>

Re: [DISCUSS] Autoformat python code with Black

problem with Python because the linter has > more rules than Checkstyle had. > > -Max > > On 29.05.19 10:16, Ismaël Mejía wrote: > >> My concerns are: > >> - The product is clearly marked as beta with a big warning. > >> - It looks like mostly a single per

1 Million Lines of Code (1 MLOC)

I was checking some metrics in our codebase and found by chance that we have passed the 1 million lines of code (MLOC). Of course lines of code may not matter much but anyway it is interesting to see the size of our project at this moment. This is the detailed information returned by loc [1]: ---

Plan for dropping python 2 support

Python 2 won't be maintained after 2020 [1]. I was wondering what will be our (Beam) plan for this. Other projects [2] have started to alert users that support will be removed so maybe we should decide or policy for this too. [1] https://pythonclock.org/ [2] https://spark.apache.org/news/plan-for-

Re: [VOTE] Release 2.13.0, release candidate #2

I don't think we need anything fancier or marking even as Blocker some of this stuff, would not be enough just to monitor that [1] has no issues? (of course if the interested party has not put the fix version to the current ongoing vote one this is a mistake). [1] https://issues.apache.org/jira/i

Re: [PROPOSAL] Preparing for Beam 2.14.0 release

Are you planning to release also the docker images that were postponed in the previous release? If so probably starting early to define that part of the process will be a good idea. On Thu, Jun 6, 2019, 7:06 PM Jean-Baptiste Onofré wrote: > +1 > > Regards > JB > Le 6 juin 2019, à 19:02, Ankur G

Re: Help triaging Jira issues

I took a look and reduced the untriaged issues to around 100. I noticed however some patterns that are producing more untriaged issues that we should have. Those can be probably automated (if JIRA has ways to do it): 1. Issues created and assigned on creation can be marked as open. 2. Once an issu

Re: Removing shading by default within BeamModulePlugin.groovy

This is fantastic. Took a look at the PR and did not see anything that jump to my eyes and also validated with two external projects with today’s snapshots (after merge) without issues so far. Great that we finally tackle this on, thanks Luke! Have one minor comment because the title of the thread

Re: learning kata as root project

Now that this is merged into master and the post is published, maybe we should make it more visible in the website. Maybe add it in https://beam.apache.org/documentation/resources/learning-resources/ and fix navigation to make it more visible too. Volunteers? On Sat, May 25, 2019 at 1:10 AM Kennet

Re: Help triaging Jira issues

. > > Kenn > > On Fri, Jun 7, 2019 at 12:53 AM Ismaël Mejía wrote: >> >> I took a look and reduced the untriaged issues to around 100. I >> noticed however some patterns that are producing more untriaged issues >> that we should have. Those can be probably

Re: Design Proposal for Cost Estimation

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Fri, Jun 7, 2019 at 10:44 PM Kenneth Knowles wrote: > > Thanks for the doc. This is really clear and readable. It all looks like a > good improvement, whatever the result of the vario

Re: Beam local website development environment

Can you give me your cwiki user id? I found three differents Rakesh Kumar there so not sure which one to assign the permissions to. On Wed, Jun 12, 2019 at 1:17 AM Rakesh Kumar wrote: > > It seems command mentioned on the confluence is little old. > The update command should be: > > ./gradlew :we

Re: Hazelcast Jet Runner

New features that we think might not yet be stable. >>>> >>>> This dual usage leads to a lot of confusion IMO. The fact that we tend to >>>> forget to remove the @Experimental tag also makes it somewhat useless. >>>> Many APIs that have been in place for

Re: Plans for Python type hints

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Fri, May 10, 2019 at 3:04 AM Ahmet Altay wrote: > > Thank you Udi. This is very helpful. I added a few comments. Overall proposed > changes make sense to me. > > From: Udi Meiri > Da

Re: Streaming pipelines in all SDKs!

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Fri, May 10, 2019 at 11:50 AM Maximilian Michels wrote: > > > So, FlinkRunner has some sort of special support for executing > > UnboundedSource via the runner in the portable world ?

Re: Artifact staging in cross-language pipelines

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Wed, May 8, 2019 at 7:29 PM Chamikara Jayalath wrote: > > > > On Tue, May 7, 2019 at 10:21 AM Maximilian Michels wrote: >> >> Here's the first draft: >> https://docs.google.com/docume

Re: Python dependency compatibility badges

+1 looks nice and can motivate people to tackle pending upgrades. On Wed, Jun 12, 2019 at 12:30 PM Robert Bradshaw wrote: > > We have a pull request to add Pypi dependency compatibility badges to > our readme: https://github.com/apache/beam/pull/8791 This looks > generally useful, though highligh

Re: Streaming pipelines in all SDKs!

Thanks On Thu, Jun 13, 2019 at 2:02 PM Łukasz Gajowy wrote: > > Created a PR: https://github.com/apache/beam/pull/8846 > > śr., 12 cze 2019 o 11:40 Ismaël Mejía napisał(a): >> >> Can you please add this to the design documents webpage. >> https://beam.apache.o

Re: [discuss] A tweak to the Python API for SDF?

:44 PM Pablo Estrada wrote: > > I have no objections. > > +Ismaël Mejía who has familiarity and interest in Java SDF. > > On Wed, Jun 5, 2019 at 11:31 AM Brian Hulette wrote: >> >> Just wanted to resurrect this to say that it seems appropriate to make the >&g

Re: [Final Reminder] Beam 2.14 release branch will be cut tomorrow at 6pm UTC

Cutting the next release branch is not equal to starting the release vote. In the past we have cut the branch even if there are still open issues and then give people some days to trim their issues. So the release manager should create the release branch in the specified date and sync with the peo

Re: [VOTE] Release vendored artifacts upgrading Guava usage to 26.0-jre, release candidate #1

The upgrade is worth for security reasons and to catch the gRPC improvements so really nice that this is happening. However I am not clear if we are mixing two things here. (1) The release of the vendored versions and (2) the upgrade of it into Beam. I think those should be separate (obviously kno

Re: [POPOSAL] Integrate BigQuery-compatible HyperLogLog algorithm into Beam

Thanks for bringing this Robin, Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ Let some comments in the doc, It is great that this is finally open and even better that it becomes part of Beam. I am not sure if this feature should go

Question about Python DirectRunner performance improvements

I stumbled recently into BEAM-3644 [1]. This issue mentions that Python direct runner saw a great performance gain because of relying on portability’s FnApiRunner. This seems to me a bit contra-intuitive considering the extra overhead of portability. How is this possible or what is the explanation

Re: Question about Python DirectRunner performance improvements

Thanks a lot for the explanation Robert, it makes sense now. On Tue, Jun 25, 2019 at 2:58 PM Robert Bradshaw wrote: > > On Tue, Jun 25, 2019 at 2:26 PM Ismaël Mejía wrote: > > > > I stumbled recently into BEAM-3644 [1]. This issue mentions that > > Python direct runner

Return types of Write transforms (aka best way to signal)

Beam introduced in version 2.4.0 the Wait transform to delay processing of each window in a PCollection until signaled. This opened new interesting patterns for example writing to a database and when ‘fully’ done write to another database. To support this pattern an IO connector Write transform mu

Re: Return types of Write transforms (aka best way to signal)

a > >>> forward-compatible way. Technically (4) could extend (3) if one wants > >>> to migrate from (3) to (4) to provide a nicer API in the future. (As > >>> an aside, it would be interesting if any of the schema work that lets > >>> us get r

Re: contributor permission for Beam Jira tickets

Done, welcome! On Mon, Jul 1, 2019 at 12:52 AM Tianyang Hu wrote: > > Hi Beam devs, > > This is Tianyang from Google. I'm closely working with Robert Burke on Beam > Go SDK. > > Can I get the contributor permission for Beam Jira tickets? My Jira ID is > htyleo. > > Thanks, > Tianyang Hu

Re: [Python] Read Hadoop Sequence File?

(Adding dev@ and Solomon Duskis to the discussion) I was not aware of these thanks for sharing David. Definitely it would be a great addition if we could have those donated as an extension in the Beam side. We can even evolve them in the future to be more FileIO like. Any chance this can happen? M

Re: Stop using Perfkit Benchmarker tool in all tests?

+1 to remove Perfkit if we can cover what we need without it. One less tool to 'learn/understand/maintain' is always good. On Fri, Jun 28, 2019 at 5:31 PM Lukasz Cwik wrote: > > +1 for removing tests that are not maintained. > > Are there features in Perfkit that we would like to be using that we

Re: WebSocket/Https connector for Apache Beam (Java)?

I was not aware of that websocket work thanks for sharing Alexey. I-Feng a Websocket based IO would be a really nice contribution so worth to bring it to the project if it is in your plans. Maybe worth to sync with JB on that. If by any chance you or any other person is interested in adding an IO

Re: [Python] Read Hadoop Sequence File?

ow to make them coexist with HadoopFormatIO > though. > > > On Tue, Jul 2, 2019 at 10:55 AM Solomon Duskis wrote: >> >> +Igor Bernstein who wrote the Cloud Bigtable Sequence File classes. >> >> Solomon Duskis | Google Cloud clients | sdus...@google.com | 914-46

Re: PR#6675 Updates

This is a holiday week in the US and a good chunk of the people in the project have been busy between Beam summit and other events in the last days, this is why reviews are taking longer than expected. Sorry, next week most things will be back to normal (hopefully). On Fri, Jul 5, 2019 at 10:27 A

Re: [VOTE] Vendored dependencies release process

+1 On Tue, Jul 9, 2019 at 10:53 PM Pablo Estrada wrote: > > Thanks Luke for writing a thorough guide for this. > +1 > > On Tue, Jul 9, 2019 at 1:27 PM Maximilian Michels wrote: >> >> +1 >> >> On 09.07.19 22:16, Udi Meiri wrote: >> > +1 LGTM >> > >> > On Mon, Jul 8, 2019 at 4:54 PM Lukasz Cwik >

Re: Hazelcast Jet Runner

to the user to decide (it might be helpful to >> list the inception date of each runner). That said, I value your concern >> Kenn. I can see that we establish a consistent onboarding of new runners >> which may involve marking them experimental for a while. >> >> -Max &

Re: Return types of Write transforms (aka best way to signal)

ts contents. >>> >>> >>> >>> >>> >>> I would probably lean towards (4) for anything that would want to >>> >>> return multiple signals/outputs (e.g. successful vs. failed writes) >>> >>> and view (3) as being a &

Re: [VOTE] Vendored Dependencies Release

+1 Run build and used diffoscope [1] to compare the generated vs staged files. We should probably in the future make the full gradle build reproducible to make this comparison trivial (a simple diff). [1] https://diffoscope.org/ On Tue, Jul 16, 2019 at 2:18 AM Lukasz Cwik wrote: > > +1 > > On M

Re: [ANNOUNCE] New committer: Robert Burke

Congrats Robert! On Tue, Jul 16, 2019 at 8:19 PM Yichi Zhang wrote: > > Congratulations! > > On Tue, Jul 16, 2019 at 10:51 AM Holden Karau wrote: >> >> Congratulations! :) >> >> On Tue, Jul 16, 2019 at 10:50 AM Mikhail Gryzykhin wrote: >>> >>> Congratulations! >>> >>> On Tue, Jul 16, 2019 at 1

Re: Using the BigQuery Storage API

This document and other developer related guides are now maintained in the Beam wiki: https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE On Wed, Jul 17, 2019 at 12:40 AM Rui Wang wrote: > > Hi, > > I have a fresh repo cloned and switch to release-2.13.0. I tried to add > "impor

Re: [discuss] A tweak to the Python API for SDF?

Is there a JIRA already to track this? On Fri, Jun 14, 2019 at 11:52 PM Ismaël Mejía wrote: > > +1 for the change for Java too both for consistency with Python and > with the way State/Timers work too. > > It would be really nice if possible to see a concrete proposed example &g

Re: [SQL] Let's split the TableProvider

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Wed, Jun 19, 2019 at 6:36 AM Kenneth Knowles wrote: > > Nice doc. I really appreciate how it gives an overview of the code. > > Kenn > > On Fri, Jun 14, 2019 at 4:14 PM Anton Kedin

Re: New Design Doc for Cost Based Optimization

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Mon, Jul 15, 2019 at 12:19 AM Rui Wang wrote: > > Thanks for your docs, Alireza! I especially appreciate you list good > references to help readers be on the same page with you (thos

Re: [Discuss] Retractions in Beam

Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/ On Wed, Jul 10, 2019 at 7:59 PM Rui Wang wrote: > > Hi Community, > > Retractions is a part of core Beam model [1]. I come up with a doc to discuss > retractions about use cases, model

Re: pubsub -> IO

Yes this is done in multiple IOs now, you can see how it is done in JdbcIO or a simpler but without explicit ReadAll transform in SolrIO. Notice that this change requires a refactor of the IO to avoid code repetition. I filled https://issues.apache.org/jira/browse/BEAM-7769 and assigned it to you,

Re: pubsub -> IO

Just discovered that RedisIO exposes ReadAll too so you can take a look at that one too. On Thu, Jul 18, 2019 at 11:39 AM Ismaël Mejía wrote: > > Yes this is done in multiple IOs now, you can see how it is done in > JdbcIO or a simpler but without explicit ReadAll transform in SolrIO.

Re: How to run DynamoDBIOTest?

This looks weird, I run the build in my machine (Ubuntu linux 18.04 + OpenJDK 1.8.0_201) + docker 18.09.8 in both master and the release 2.14.0 branch and it executes without issue. This test uses a docker image as a sort of embedded server to simulate the Amazon backend (localstack). All builds we

Re: How to run DynamoDBIOTest?

Created https://issues.apache.org/jira/browse/BEAM-7794 to track this issue, any extra info is welcomed in the meantime maybe a good idea to disable the test. On Mon, Jul 22, 2019 at 3:44 PM Elliotte Rusty Harold wrote: > > FYI, I killed the job this morning after letting it run over the > weeknd

How to expose/use the External transform on Java SDK

After Beam Summit EU I was curious about the External transform. I was interested on the scenario of using it to call python code in the middle of a Java pipeline. This is a potentially useful scenario for example to evaluate models from python ML frameworks on Java pipelines. In my example I did a

Re: How to expose/use the External transform on Java SDK

unner, not for cross-language use-cases. Is that correct ? May be > >> we can reuse it for latter as well. > >> > >> Thanks, > >> Cham > >> > >> [a] > >> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/ma

Re: [VOTE] Release 2.14.0, release candidate #1

To complete the release we need to have at least three +1 binding votes (votes from PMC members) as stated in [1]. So far we have only 2. Thomas (and the others). The blog post PR is now open [2] please help us add missing features or maybe to highlight the ones you consider important in the PR co

[ANNOUNCE] New committer: Jan Lukavský

Hi, Please join me and the rest of the Beam PMC in welcoming a new committer: Jan Lukavský. Jan has been contributing to Beam for a while, he was part of the team that contributed the Euphoria DSL extension, and he has done interesting improvements for the Spark and Direct runner. He has also bee

Re: [VOTE] Release 2.14.0, release candidate #1

Oups Robert pointed to me that I have probably not counted correctly. There were indeed already 3 PMC +1 votes. Pablo, Robert and Ahmet. Please excuse me for the extra noise. On Wed, Jul 31, 2019 at 9:46 AM Ismaël Mejía wrote: > > To complete the release we need to have at least three +1 b

Snapshots not been updated

Hello, It seems the 2.16.0-SNAPSHOTS are not been updated since last friday 02/08. https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.16.0-SNAPSHOT/ Can somebody PTAL. Thanks

  1   2   3   4   5   6   7   8   >