rings additional
> complexity into pipeline deployment and performance overhead.
>
> Perhaps, it will be changed in long term, but for now, I may guess that
> the most of Beam pipelines still use the same SDK IO connectors as a
> pipeline itself.
>
> —
> Alexey
>
> On
I think the idea of cross language is that an IO is only in one language
and others can use that IO. My feeling is that the idea of “what language
is this IO in” becomes an implementation detail that folks won’t have to
care about longer term. There are enhancements needed to the expansion
service
These are great. Thank you so much!
On Mon, Jan 9, 2023 at 6:33 AM Alexey Romanenko
wrote:
> Always happy to help!
>
> Many thanks for your work to make Beam website better!
>
> —
> Alexey
>
> On 6 Jan 2023, at 21:54, Alex Kosolapov
> wrote:
>
> Thank you, Ahmet! Happy to help! Both changes
Given the increasing importance of multi language pipelines, it does seem
that we should expand the capabilities of the DirectRunner or just go all
in on FlinkRunner for testing and local / small scale development
On Wed, Dec 28, 2022 at 12:47 AM Robert Burke wrote:
> Probably either on Flink,
outputs by Foo.output1, Foo.output2, etc.
>>>
>>> > Multiple inputs are merged with a Flatten transform.
>>>
>>> PTransfoms can have named inputs as well (they're not always
>>> symmetric), so I let inputs be a map if they care to distinguish them.
>>>
>>>
To build on Kenn's point, if we leverage existing stuff like dbt we get
access to a ready made community which can help drive both adoption and
incremental innovation by bringing more folks to Beam
On Wed, Dec 14, 2022 at 2:57 PM Kenneth Knowles wrote:
> 1. I love the idea. Back in the early
I strongly believe that we should continue to have Beam optimize for the
user - and while having separate components would allow those of us who are
contributors and committers move faster, the downsides of not having
everything "in one box" for a new user where the components are all
relatively
It would be helpful to explain the scope here - if the previous iteration
was too overweight, it would be good to be intentional.
I think all would agree that being more prescriptive would help IO makers
(especially those from startups looking to expand their reach).
On Mon, Dec 12, 2022 at 7:32
This is a good heads up, thank you Cristian.
On Thu, Dec 1, 2022 at 8:13 AM Cristian Constantinescu
wrote:
> Hi,
>
> I came across some Kafka info and would like to share for those
> unaware. Kafka is planning to drop support for Java 8 in Kafka 4 (Java
> 8 is deprecated in Kafka 3), see
Would it be helpful to add these answers to the Beam docs?
On Mon, Nov 14, 2022 at 4:35 AM Jan Lukavský wrote:
> I somehow missed these answers, Reuven and Kenn, thanks for the
> discussion, it helped me clarify my understanding.
>
> Jan
> On 10/26/22 21:10, Kenneth Knowles wrote:
>
>
>
> On
This is super interesting, thank you Damon!
On Thu, Nov 10, 2022 at 10:51 AM Damon Douglas via dev
wrote:
> Hello Everyone,
>
> I created https://github.com/apache/beam/pull/24081 to start a
> conversation around WebAssembly support in Beam.
>
> WebAssembly is an experimental technology.
Congratulations Yi!
On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles wrote:
> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Yi Hu (y...@apache.org)
>
> Yi started contributing to Beam in early 2022. Yi's contributions are very
> diverse! I/Os,
Hey folks,
I've found myself repeatedly being very untimely in providing reviews on
PRs where I've been added as a reviewer. (Mea culpa and thank you for your
understanding to those who have tagged me and emailed me to nudge me along.)
Does anyone have any great tips about how to be super on
Congrats Ritesh!
On Thu, Nov 3, 2022 at 4:16 PM Kenneth Knowles wrote:
> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new committer:
> Ritesh Ghorse (riteshgho...@apache.org)
>
> Ritesh started contributing to Beam in mid-2021 and has contributed
> immensely to bringin
I think this is a great idea - making any many existing IOs as possible
available to developers in any language is a huge win (and helps reduce the
need to re-implement IOs on a language-by-language basis going forward).
On Thu, Nov 3, 2022 at 11:25 AM Ahmed Abualsaud via dev
wrote:
> Hi all,
>
s been
> superseded by https://beam.apache.org/documentation/io/connectors/
>
> Would there be any objections to that?
>
> On Thu, Oct 27, 2022 at 2:04 PM Sachin Agarwal via dev <
> dev@beam.apache.org> wrote:
>
>> JDBCIO is available as a Java-based IO. It is also listed on
&
JDBCIO is available as a Java-based IO. It is also listed on
https://beam.apache.org/documentation/io/connectors/
On Thu, Oct 27, 2022 at 2:01 PM Charles Kangai
wrote:
> What about jdbc?
> I want to use Beam to read/write to/from a relational database, e.g.
> Oracle or Microsoft SQL Server.
>
It turns out there was a commit submitted here!
https://github.com/nanhu-lab/beam/commit/d4f5fa4c41602b4696737929dd1bdd5ae2302a65
Related GH issue: https://github.com/apache/beam/issues/23074
On Tue, Aug 30, 2022 at 10:28 AM Sachin Agarwal wrote:
> I would posit that something is better than
This is wonderful - thank you so much to you and the whole Talend team to
make Beam better!
On Fri, Sep 16, 2022 at 9:11 AM Alexey Romanenko
wrote:
> Hi everybody,
>
> As some of you may know, at Talend, we’ve been working for a while to add
> TPC-DS benchmark suite into Beam. We believe that
Andrew,
Thanks so much for the feedback and glad the getting started materials are
helping.
Would you like a downloadable container that works out of the box for your
local machine or to spin up on AWS or GCP or DO or something like that?
Just trying to make sure we understand the gap correctly
I would posit that something is better than nothing - did we ever see that
generic implementation?
On Tue, Aug 30, 2022 at 10:22 AM Austin Bennett
wrote:
> Is there enough commonality across Delta, Hudi, Iceberg for this generic
> solution? I imagined we'd potentially have individual IOs for
This is wonderful to hear -
https://beam.apache.org/contribute/get-started-contributing/#contribute-code
has the process to contribute; we're very much looking forward to seeing
your DataLakeIO!
On Fri, Aug 5, 2022 at 9:02 AM 张涛 wrote:
>
> Hi, we developed a new IO connector named DataLakeIO,
On Tue, Jul 26, 2022 at 6:12 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:
>
>
> On Mon, Jul 25, 2022 at 12:53 PM Lina Mårtensson via dev <
> dev@beam.apache.org> wrote:
>
>> Hi dev,
>>
>> We're starting to incorporate BigTable in our stack and I've delighted
>> my co-workers with
23 matches
Mail list logo
