Re: Re: [YAML] ReadFromKafka with yaml

To use "ReadFromKafka" from Flink, you additionally need to specify pipeline option "--experiments=use_deprecated_read" I believe. This is due to a known issue: https://github.com/apache/beam/issues/20979 Thanks, Cham On Wed, Jan 10, 2024 at 9:56 PM Yarden BenMoshe wrote: > Thanks for the

RE: Re: [YAML] ReadFromKafka with yaml

Thanks for the detailed answer. I forgot to mention that I am using FlinkRunner as my Setup. Will this work with this runner as well? On 2024/01/10 13:34:28 Ferran Fernández Garrido wrote: > Hi Yarden, > > If you are using Dataflow as a runner, you can already use > ReadFromKafka (introduced

RE: Re: KafkaIO does not make use of Kafka Consumer Groups [kafka] [java] [io]

Can't agree with Shahar Frank more On 2023/04/19 18:17:15 Shahar Frank wrote: > Hi Daniel, > > I think I've already answered these in a previous email but let me answer > them again. > > I was specifically responding to quoted points from your last email. I >> really don't understand why you,

RE: Re: Project Proposal

On 2023/03/23 15:00:25 Anand Inguva via dev wrote: > Hi, > > Thanks for the proposal. Can you share the google doc link for your > proposal? It would be easier to go back and forth on reviews. > > I am happy to review it and provide feedback on it. > > Thanks, > Anand > > On Sun, Mar 19, 2023 at

RE: Re: Project Proposal

On 2023/03/23 15:00:25 Anand Inguva via dev wrote: > Hi, > > Thanks for the proposal. Can you share the google doc link for your > proposal? It would be easier to go back and forth on reviews. > > I am happy to review it and provide feedback on it. > > Thanks, > Anand > > On Sun, Mar 19, 2023 at

Re: Re: Re: RE: Re: unvendoring bytebuddy

Thanks for doing so much work verifying this and analyzing it! It really seems like we did all this for mockito, so it has very little risk of impacting users. And you've verified it is working with mockito now. So I think I'm in favor of unvendoring. This will make it much easier to get bugfixes,

Re: Re: RE: Re: unvendoring bytebuddy

Hmm. Too bad the information on the jira is inadequate to explain or justify the change. TBH if faced with a conflict between bytebuddy and mockito, working to use mocks less, or in more straightforward ways, would have been my preference. This isn't actually a diamond dep problem that impacts

RE: Re: Re: Re: Re: [Question][Contribution] Python SDK ByteKeyRange

LexicographicKeyRangeTracker supports both string and byte keys so it’s more complex than tracker that would only support byte keys. This is why I would make ByteKeyRestrictionTracker and if someone wants to support string keys they could make another contribution. On 2022/02/15 22:17:37

RE: Re: Re: Re: [Question][Contribution] Python SDK ByteKeyRange

That tracker is not a restriction tracker which I need for my Bigtable reader SDF. When I started working on this tracker I noticed that it was implemented in Java and I figured it would be best to make functionally similar implementation in Python. LexicographicKeyRangeTracker is not that

RE: Re: Contributor permission for Jira tickets

My username is samnisol. On 2022/02/15 18:52:33 Ahmet Altay wrote: > What is your jira username? > > On Tue, Feb 15, 2022 at 2:12 AM Sami Niemi > mailto:sa...@solita.fi>> wrote: > > > Hello, > > > > > > > > This is Sami from Solita. I’m working on ByteKeyRange and > > ByteKeyRestrictionTracker

RE: Re: Re: [Question][Contribution] Python SDK ByteKeyRange

Hi Ismaël, What I’ve currently been working on locally is almost 100% based on that Java implementation. I suppose I need to create Jira issue and make the contribution. On 2022/02/15 09:19:33 Ismaël Mejía wrote: > Oh, forgot to add also the link to the tests that cover most of those >

Re: Re: [Question][Contribution] Python SDK ByteKeyRange

Oh, forgot to add also the link to the tests that cover most of those unexpected cases: [2] https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java On Tue, Feb 15, 2022 at 10:17 AM Ismaël Mejía wrote: >

Re: Re: [Question][Contribution] Python SDK ByteKeyRange

Great idea, please take a look at the Java ByteKeyRestrictionTracker implementation for consistency [1] I remember we had to deal with lots of corner cases so probably worth a look. [1]

RE: Re: [Question][Contribution] Python SDK ByteKeyRange

Hello Robert, Beam has documented only OffsetRangeTracker [1] for new SDF API. Since Beam is moving away from Source API, I thought it would be nice to develop IO connectors by using new SDFs. For this I need to create restriction tracker that follows new SDF API. So I propose adding

RE: Re: Question about Go SDK Built-in I/O Transforms

Hi Robert, Thanks for your reply and sorry about my delay, I forgot to subscribe to dev list too, my fault. :( We are really excited to know that we can write the Built-in I/O Transforms using xlang and Splittable DoFns. About our use cases: - A large part of the company uses Go and has a lot

Re: Re:

Hello Pablo! The "JdbcIO.Write" allows you to write rows without a statement or statement preparer, but not all functionality works without them. The method "WithResults" requires a statement and statement preparer. And also the ticket and "//

Re: Re-running GitHub Actions jobs

There's a "Re-run Jobs" button at the top right when you open up one of the jobs: [image: image.png] On Thu, Sep 3, 2020 at 12:02 PM Heejong Lee wrote: > > > On Thu, Sep 3, 2020 at 11:05 AM Brian Hulette wrote: > >> The new GitHub Actions workflows that run Java and Python tests against >>

Re: Re-running GitHub Actions jobs

On Thu, Sep 3, 2020 at 11:05 AM Brian Hulette wrote: > The new GitHub Actions workflows that run Java and Python tests against > different targets (macos, ubuntu, windows) are great! But just like our > Jenkins infra they flake occasionally. Should we be re-running all of these > jobs until we

RE: Re: [RESULT] [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

Hey Beam Team, Thanks for your support around this! I will be be submitting an individual contributors license agreement. http://www.apache.org/licenses/contributor-agreements.html Cheers! Julian On 2020/01/16 19:17:24 Aizhamal Nurmamat kyzy wrote: > I was going to let Julian answer as he

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Thanks for sharing those results. The second set (executors at 20-30) look similar to what I would have expected. BEAM-5036 definitely plays a part here as the data is not moved on HDFS efficiently (fix in PR awaiting review now [1]). To give an idea of the impact, here are some numbers from my

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Something here on the Beam side is clearly linear in the input size, as if there's a bottleneck where were' not able to get any parallelization. Is the spark variant running in parallel? On Fri, Sep 28, 2018 at 4:57 AM devinduan(段丁瑞) wrote: > Hi > I have completed my test. > 1. Spark

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Got it. I will also set "spark.dynamicAllocation.enabled=true" to test. From: Tim Robertson<mailto:timrobertson...@gmail.com> Date: 2018-09-19 17:04 To: dev@beam.apache.org<mailto:dev@beam.apache.org> CC: j...@nanthrax.net<mailto:j...@nanthrax.net> Subj

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Thank you Devin Can you also please try Beam with more spark executors if you are able? On Wed, Sep 19, 2018 at 10:47 AM devinduan(段丁瑞) wrote: > Thanks for your help! > I will test other examples of Beam On Spark in the future and then feed > back the results. > Regards > devin > > > *From:*

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Thanks for your help! I will test other examples of Beam On Spark in the future and then feed back the results. Regards devin From: Jean-Baptiste Onofré Date: 2018-09-19 16:32 To: devinduan(段丁瑞); dev Subject: