Re: Performance and Cost benchmarking

2022-09-26 Thread Andrew Pilloud via dev
Hi Pranav, I left some comments on your design. Your doc discusses a bunch of details about infrastructure such as testing frameworks, automation, and performance databases, but doesn't describe how it will fit in with our existing infrastructure (Load Tests, Nexmark, Jenkins, InfluxDB, Grafina).

Re: [idea] A new IO connector named DataLakeIO, which support to connect Beam and data lake, such as Delta Lake, Apache Hudi, Apache iceberg.

2022-09-26 Thread Sachin Agarwal via dev
It turns out there was a commit submitted here! https://github.com/nanhu-lab/beam/commit/d4f5fa4c41602b4696737929dd1bdd5ae2302a65 Related GH issue: https://github.com/apache/beam/issues/23074 On Tue, Aug 30, 2022 at 10:28 AM Sachin Agarwal wrote: > I would posit that something is better than

Performance and Cost benchmarking

2022-09-26 Thread Pranav Bhandari
Hello, Hope this email finds you well. I have attached a link to a doc which discusses the design for a performance and cost benchmarking framework to be used by Beam IOs and Google-provided dataflow templates. Please feel free to comment on the doc with any questions, concerns or ideas you

Beam High Priority Issue Report (71)

2022-09-26 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/23350 [Bug]:

Re: [JmsIO] => Pull Request to fix message acknowledgement issue

2022-09-26 Thread BALLADA Vincent via dev
De : Luke Cwik Date : jeudi, 8 septembre 2022 à 19:17 À : BALLADA Vincent Cc : dev@beam.apache.org Objet : Re: [JmsIO] => Pull Request to fix message acknowledgement issue [vwP6KQExYeP8ewASUVORK5CYII=] [EXT] Could we have more than one active checkpoint per reader instance? Yes. Readers