Very nice resource, thanx Mikhail.
On Fri, 9 Aug 2019 at 05:22, Mikhail Gryzykhin wrote:
> Unfortunately no, I don't have those for streaming explicitly.
>
> However most of code is shared between streaming and batch with main
> difference in initialization. Same goes for boilerplate parts of
Commenting here vs. on the PR since related to the overall approach.
Wouldn't it be simpler to have the runner just track a unique ID for each
worker and use that to communicate if the cache is valid or not?
* When the bundle is started, the runner tells the worker if the cache has
become
I think this is a great basis for guidelines on the blog post at a minimum.
A real changelog, gathered in one place on the web page, is more useful
than scanning the blog.
The Jira version summaries are pretty close, but we would need to be much
more serious about making good titles and getting
Hi Robert,
Thanks for your information, that explains the behavior I noticed. I guess
my current solution would be somehow to shift the watermark or start the
streaming process before any files come in to settle down the initial
watermark.
I will keep watching the JIRA you shared, thanks for the
If somebody is using JPMS and they attempt to import beam, they get a
compile time error. Some other projects I work on have been getting
user reports about this.
See
https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-19.md
for more details.
On
On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold
wrote:
>
>
> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía wrote:
>
>> a per case approach (the exception could be portable runners not based on
>> Java).
>>
>> Of course other definitions of being Java 11 compatible are interesting
>> but
+1
I've installed from the source in apache/dist.
I've run unit tests in Python 3.6, and wordcount in Python 3.6 in Direct
and Dataflow runners.
Thanks!
-P.
On Tue, Aug 20, 2019 at 11:41 AM Hannah Jiang
wrote:
> Yes, I agree this is a separate topic and shouldn't block 2.15 release.
> There
On Tue, Aug 20, 2019 at 1:41 PM Robert Bradshaw wrote:
> On Mon, Aug 19, 2019 at 5:44 PM Ahmet Altay wrote:
> >
> >
> >
> > On Mon, Aug 19, 2019 at 9:56 AM Brian Hulette
> wrote:
> >>
> >>
> >>
> >> On Fri, Aug 16, 2019 at 5:17 PM Chad Dombrova
> wrote:
>
> >> Agreed on float since
I have run the seed job. Fortunately the new precommit is only triggered by
SQL related PRs. I can follow up on those PRs if there is any negative
feedback.
But thanks for the heads up.
-Rui
On Tue, Aug 20, 2019 at 2:58 PM Boyuan Zhang wrote:
> And after you run seed job, other PR's commit
And after you run seed job, other PR's commit may also trigger your
percommit test. You can set your precommit task as only triggered by phrase
for test purpose.
Thank you Yifan!
-Rui
On Tue, Aug 20, 2019 at 2:52 PM Yifan Zou wrote:
> Run the seed job then trigger your tests by using phrase.
>
> On Tue, Aug 20, 2019 at 2:39 PM Rui Wang wrote:
>
>> Hi Community,
>>
>> I am trying to add a new precommit task (see [1] and [2]), and the PR is
>> pending.
Run the seed job then trigger your tests by using phrase.
On Tue, Aug 20, 2019 at 2:39 PM Rui Wang wrote:
> Hi Community,
>
> I am trying to add a new precommit task (see [1] and [2]), and the PR is
> pending. Does anyone know how to test the added precommit directly in the
> PR before merging
Hi Community,
I am trying to add a new precommit task (see [1] and [2]), and the PR is
pending. Does anyone know how to test the added precommit directly in the
PR before merging it?
[1]:
https://github.com/apache/beam/pull/9210/files#diff-d6dfd4f4d675cfe2d6f52ae6fea472d0
[2]:
The original timestamps are probably being assigned in the
watchForNewFiles transform, which is also setting the watermark:
https://github.com/apache/beam/blob/release-2.15.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L668
Until https://issues.apache.org/jira/browse/BEAM-644
On Mon, Aug 19, 2019 at 5:44 PM Ahmet Altay wrote:
>
>
>
> On Mon, Aug 19, 2019 at 9:56 AM Brian Hulette wrote:
>>
>>
>>
>> On Fri, Aug 16, 2019 at 5:17 PM Chad Dombrova wrote:
>> Agreed on float since it seems to trivially map to a double, but I’m
>> torn on int still. While I
> The issue is also tracked here:
>
https://jira.apache.org/jira/browse/BEAM-7870 There are some suggestions
> in the issue. I think the best solution is to allow execution of the
> source API parts of KafkaIO/PubSubIO (on the Runner) and the following
> UDFs (in the environment). Since those do
Yes, I agree this is a separate topic and shouldn't block 2.15 release.
There is already a JIRA ticket, I will update it with more details.
On Tue, Aug 20, 2019 at 11:32 AM Ahmet Altay wrote:
>
>
> On Tue, Aug 20, 2019 at 10:18 AM Yifan Zou wrote:
>
>> Hi all,
>>
>> This is a friendly
On Tue, Aug 20, 2019 at 10:18 AM Yifan Zou wrote:
> Hi all,
>
> This is a friendly reminder. Please help to review, verify and vote on
> the release candidate #2 for the version 2.15.0.
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> I've
Ah this is a great feature. Thanks for looking into it!
On Tue, Aug 20, 2019 at 12:44 AM Ismaël Mejía wrote:
> Hello Cannan,
>
> Welcome! You were added to the contributors role and the ticket was
> assigned to you too. Now you can also self assign JIRAs if you want to
> contribute in other
On Tue, Aug 20, 2019 at 4:29 AM Maximilian Michels wrote:
> Hi Chad!
>
> Thank you so much for your feedback. You are 100% on the right track.
> What you are seeing is a core issue that also needs to be solved for
> KafkaIO to be fully usable in other SDKs. I haven't had much time to
> work on
Hi all,
This is a friendly reminder. Please help to review, verify and vote on the
release candidate #2 for the version 2.15.0.
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
I've verified Java quickstart & mobile games, and Python (both tar and
On Tue, Aug 20, 2019 at 8:56 AM Lukasz Cwik wrote:
>
>
> On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay wrote:
>
>>
>>
>> On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise wrote:
>>
>>> There is a PR open for this: https://github.com/apache/beam/pull/9331
>>>
>>> (it wasn't tagged with the JIRA and
A side note about this test:
Now we only have py2 and py35, so it only fails with py35. I am introducing
minor versions, which will add py36 and py37, and all py3 are flaky.
It's really difficult to pass Portable Precommit with minor versions, the
chance of passing the test is around 15%.
On Mon,
On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay wrote:
>
>
> On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise wrote:
>
>> There is a PR open for this: https://github.com/apache/beam/pull/9331
>>
>> (it wasn't tagged with the JIRA and therefore not linked)
>>
>> I think it is worthwhile to explore how
On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía wrote:
> a per case approach (the exception could be portable runners not based on
> Java).
>
> Of course other definitions of being Java 11 compatible are interesting
> but probably not part of our current scope. Actions like change the
> codebase to
Many different people understand different things for Java 11 compatibility
and probably the easiest path for us is to define exactly what we (Beam)
meant with being Java 11 compatible.
The definition that Michał gave seems aligned with the current scope. Beam
published artifacts compiled with
Hi Chad!
Thank you so much for your feedback. You are 100% on the right track.
What you are seeing is a core issue that also needs to be solved for
KafkaIO to be fully usable in other SDKs. I haven't had much time to
work on this in the past weeks but now is the time :)
The cross-language
Hello Cannan,
Welcome! You were added to the contributors role and the ticket was
assigned to you too. Now you can also self assign JIRAs if you want to
contribute in other areas.
On Mon, Aug 19, 2019 at 10:01 PM Canaan Silberberg wrote:
>
> HI all
>
> I'm working with beam's BigQueryIO over
28 matches
Mail list logo
