Congratulations!
On Fri, Sep 11, 2020, 5:54 PM Andrew Psaltis
wrote:
> Congrats!
>
> On Sat, Sep 12, 2020 at 7:43 AM Reza Rokni wrote:
>
>> Thanx everyone! Looking forward to being able to contribute more :-)
>>
>> On Sat, Sep 12, 2020 at 4:33 AM Valentyn Tymofieiev
>> wrote:
>>
>>>
Congrats!
On Sat, Sep 12, 2020 at 7:43 AM Reza Rokni wrote:
> Thanx everyone! Looking forward to being able to contribute more :-)
>
> On Sat, Sep 12, 2020 at 4:33 AM Valentyn Tymofieiev
> wrote:
>
>> Congrats!
>>
>> On Thu, Sep 10, 2020 at 8:08 PM Connell O'Callaghan
>> wrote:
>>
>>>
Thanx everyone! Looking forward to being able to contribute more :-)
On Sat, Sep 12, 2020 at 4:33 AM Valentyn Tymofieiev
wrote:
> Congrats!
>
> On Thu, Sep 10, 2020 at 8:08 PM Connell O'Callaghan
> wrote:
>
>> Excellent- well done Reza!!!
>>
>> On Thu, Sep 10, 2020 at 7:35 PM Austin Bennett <
On Fri, Sep 11, 2020 at 3:02 PM Robert Bradshaw wrote:
> The long term goal is for Dataflow to use the external containers rather
> than its own. Hopefully this happened sooner rather than later, and until
> then you can specify the beam container as a custom container.
>
> On Fri, Sep 11, 2020
On Fri, Sep 11, 2020 at 3:29 PM Kyle Weaver wrote:
> > Could someone create an ‘infra’ component in Jira (I don’t have access)?
>
>
> Could these issues go under the "testing" component?
>
That is an option. There are things beyond testing as well like Github
Actions, Jenkins inventory jobs,
> Could someone create an ‘infra’ component in Jira (I don’t have access)?
Could these issues go under the "testing" component?
On Fri, Sep 11, 2020 at 3:11 PM Tyson Hamilton wrote:
> Hi All,
>
> Below is a summary of the improvements made to Beam’s infrastructure in
> the last few months.
Hi All,
Below is a summary of the improvements made to Beam’s infrastructure in the
last few months. The completion of this work is thanks to Damian (damgad)
and Tobiasz (TobKed) from Polidea. Thank you!!!
-
Jenkins tests are not always triggered / Switch to new jenkins infra
-
Ok great. Next question:
What is the relationship between sdks/python/container/boot.go and
Dataflow? Is this file used within the Dataflow bootstrapping process?
We're currently investigating a switch from Flink to Dataflow, and in doing
so we hope to be able to work our way back to using
Welcome Kiley, I have done as you have requested.
On Fri, Sep 11, 2020 at 1:18 PM Kiley Sok wrote:
> Hello,
>
> I'm Kiley, a SWE at Google working on Beam. Can I be added as a
> contributor to Jira? My username is kileys.
>
> Thanks,
> Kiley
>
On Fri, Sep 11, 2020 at 2:11 PM Robert Bradshaw wrote:
> Hmm... this is a difficult question. I think adding support for a pip.conf
> probably makes the most sense, despite it being yet another option.
>
+1 - I think this is a good flag to add. I heard similar user requests for
passing specific
Hmm... this is a difficult question. I think adding support for a pip.conf
probably makes the most sense, despite it being yet another option.
Another alternative is to simply pre-install the dependencies you want (or
even just override /etc/pip.conf) in a custom container.
On Wed, Sep 9, 2020
Hello,
I'm Kiley, a SWE at Google working on Beam. Can I be added as a contributor
to Jira? My username is kileys.
Thanks,
Kiley
Yes, sure. Though, I’d prefer to keep only v2 ones after a while (several
releases) in case if there are no users objections.
> On 11 Sep 2020, at 19:41, Robert Bradshaw wrote:
>
> Makes sense to me. We could keep the v1 ones around as-is with fairly low
> cost, right?
>
> On Fri, Sep 11,
I am not deep on the details myself but have reviewed various Avro upgrade
changes such as https://github.com/apache/beam/pull/9779 and also some
internal that I cannot link to. I believe the changes are small and quite
possibly we can create sdks/java/extensions/avro that works with both Avro
1.8
> The concern here is that Avro 1.9 is not backwards compatible with Avro 1.8,
> so would the future world would not be a simple "bring your own avro" but
> might require separate dataflow-with-avro-1.8 and dataflow-with-avro-1.9
> targets which certainly isn't scalable. (Or am I mistaken here?
Dear BEAM Team,
I am trying to build and run a pipeline on Google Cloud Dataflow.
I have a situation where I need to use an event timestamp after parsing the
received custom message in DoFn. I called context.outputWithTimestamp but
it complained with exception about skew:
ERROR
What if we introduced a core-lite package without avro? (We could take
inventory and see if there are other dependencies we could/should make
optional as well.) The existing core module would remain the same, but it
would provide a way for users to use other avros with Beam.
On Fri, Sep 11, 2020
Getting Avro out of core is a good idea in principle but it has
consequences for users.
The cleanest solution implies moving the packages as done in the PR
but this will essentially break every existing user, so we should
measure the impact on users and agree if it is worth to break SDK core
Hello,
In Beam, there are two versions of AWS IO connectors for Java SDK - based on
AWS SDK v1 [1] and v2 [2]. For now, they are pretty equal in terms of
functionality, but since AWS SDK v2 is more modern (it's a major rewrite of the
version 1.x code base, it’s built on top of Java 8+ and adds
Top-post: I'm generally in favor of moving Avro out of core specifically
because it is something where different users (and dep chains) want
different versions. The pain caused by having it in core has come up a lot
to me. I don't think backwards-compatibility absolutism helps our users in
this
Robert,
Just replying to confirm that I specifically pulled v2.24.0-RC3 of the SDK and
the issue has been resolved.
Thanks Robert.
Regards,
Patrick Mitchell
On 2020/09/11 00:29:10, Robert Burke
mailto:r...@frantil.com>> wrote:
> This is a default limit on the receive Buffer on the SDK side.>
On Thu, Sep 10, 2020 at 2:48 PM Brian Hulette wrote:
>
> On Tue, Sep 8, 2020 at 9:18 AM Robert Bradshaw
> wrote:
>
>> IIRC Dataflow (and perhaps others) implicitly depend on Avro to write
>> out intermediate files (e.g. for non-shuffle Fusion breaks). Would
>> this break if we just removed it?
Hi Monica,
Done. Welcome to Beam!
Alexey
> On 11 Sep 2020, at 03:01, Monica Song wrote:
>
> Hi all--
>
> I'm Monica, a SWE at Google! Can someone add me as a contributor for Beam's
> Jira issue tracker? My username is monicadsong.
>
> I would like to create/assign tickets for my work. I
Hi everyone,
PR https://github.com/apache/beam/pull/12748 now passes all the checks, and
could potentially be merged (not advocating this, just saying). I've
rebased on the latest master as of today. I've also left a comment in the
PR with the high level changes for ALL the modules. I encourage
24 matches
Mail list logo
