Hi Luke,
Thanks a lot for your reply. Since it allows to share one SDK harness
between multiple executable stages, the control service termination may
occur much later than the completion of an executable stage. This is the
main reason I prefer runners to control the teardown of DoFns.
Regarding
Hi Noah!
Thanks for contributing this. The change is relatively small, so I think
it's best to discuss it in a pull request, so we can look at the diff.
My preliminary comment would be that the first part makes sense:
if isinstance(schema, (str, unicode)):
schema =
This seems extremely useful.
I assume you mean `@OnTimer("timers")` in your example. I would suggest
that the parameter annotation be something other than @TimerId since that
annotation is already used for a very similar but different purpose; they
are close enough that it is tempting to pun
Hi,
I am interested to contribute to this issue
https://issues.apache.org/jira/browse/BEAM-3658
I have followed the contribution guide and was able to build the project
locally using gradlew commands.
I wanted to debug and trace the issue further by running the tests locally
using Intellij
Thanks for trying this out. Yes, this is definitely something that
should be supported (and tested).
On Mon, Oct 21, 2019 at 3:40 PM Igor Durovic wrote:
>
> Hi everyone,
>
> The interactive beam example using the DirectRunner fails after execution of
> the last cell. The recursion limit is
Hi everyone,
The interactive beam example using the DirectRunner fails after execution
of the last cell. The recursion limit is exceeded during the calculation of
the cache label because of a circular reference in the PipelineInfo object.
The constructor
I have created this issue (https://issues.apache.org/jira/browse/BEAM-8452).
The contributors guide suggests that if an issue is your first, it should
be discussed on this mailing list. I would like to hear thoughts,
questions, concerns etc on this proposed fix:
def process(self, element,
Hello everybody,
Just a friendly heads up.
Seems that CWiki changed authentication approach and people with non-apache
logins might have lost rights to edit Beam pages. So if you don't see
"Edit" button, that might be the case.
For committers: use your apache ldap. For others, I guess the
BEAM-6857 documents the need for dynamic timer support in the Beam API. I
wanted to make a proposal for what this API would look like, and how to
express it in the portability protos.
Background: Today Beam (especially BeamJava) requires a ParDo to statically
declare all timers it accesses at
thanks again!
On Mon, Oct 21, 2019 at 1:03 PM Robert Bradshaw wrote:
> I just merged https://github.com/apache/beam/pull/9845 which should
> resolve the issue.
>
> On Mon, Oct 21, 2019 at 12:58 PM Chad Dombrova wrote:
> >
> > thanks!
> >
> > On Mon, Oct 21, 2019 at 12:47 PM Kyle Weaver
>
I just merged https://github.com/apache/beam/pull/9845 which should
resolve the issue.
On Mon, Oct 21, 2019 at 12:58 PM Chad Dombrova wrote:
>
> thanks!
>
> On Mon, Oct 21, 2019 at 12:47 PM Kyle Weaver wrote:
>>
>> This issue is being tracked at
>>
thanks!
On Mon, Oct 21, 2019 at 12:47 PM Kyle Weaver wrote:
> This issue is being tracked at
> https://issues.apache.org/jira/browse/BEAM-8416.
>
> On Mon, Oct 21, 2019 at 9:42 PM Chad Dombrova wrote:
>
>> Hi all,
>> Is anyone else getting these errors in
>>
This issue is being tracked at
https://issues.apache.org/jira/browse/BEAM-8416.
On Mon, Oct 21, 2019 at 9:42 PM Chad Dombrova wrote:
> Hi all,
> Is anyone else getting these errors in
> apache_beam.runners.portability.artifact_service_test.ZipFileArtifactServiceTest?
>
> They seem to be taking
Hi all,
Is anyone else getting these errors in
apache_beam.runners.portability.artifact_service_test.ZipFileArtifactServiceTest?
They seem to be taking two forms:
zipfile.BadZipFile: Bad CRC-32 for file
Yes, the test should be fixed.
On Mon, Oct 21, 2019 at 11:20 AM Jan Lukavský wrote:
>
> Hi Robert,
>
> I though it would be that case. ParDoLifecycleTest, however, does not
> currently allow for empty bundles. We have currently worked around this
> in Flink by avoiding the creation of these
Yes, please update the test.
On Mon, Oct 21, 2019 at 11:20 AM Jan Lukavský wrote:
> Hi Robert,
>
> I though it would be that case. ParDoLifecycleTest, however, does not
> currently allow for empty bundles. We have currently worked around this
> in Flink by avoiding the creation of these
Hi Robert,
I though it would be that case. ParDoLifecycleTest, however, does not
currently allow for empty bundles. We have currently worked around this
in Flink by avoiding the creation of these bundles, but maybe the test
should be modified so that it adheres to the model [1].
Jan
[1]
Approach 2 is currently the suggested approach[1] for DoFn's to shutdown.
Note that SDK harnesses can terminate instances any time they want and
start new instances anytime as well.
Why do you want to expose this logic so that Runners could control it?
1:
Yes, the model allows them.
It also takes less work to avoid them in general (e.g. imagine one
reshuffles N elements to M > N workers. A priori, one would "start" a
bundle and then try to read all data destined for that
worker--postponing this until one knows that the set of data for this
worker
Hi Max,
that is true, but then we have two orthogonal issues:
a) correctness - if empty bundles are aligned with the model, then
validates runner tests should take that into account
b) performance - that can be dealt with in separate JIRA issue, if needed
WDYT?
Jan
On 10/21/19 3:22 PM,
Do you know why an empty bundle might be created?
On Mon, Oct 21, 2019 at 1:42 PM Jan Lukavský wrote:
> Hi,
>
> when debugging a flaky ParDoLifecycleTest in FlinkRunner, I have found a
> situation, where Flink might create empty bundle - i.e. call
> @StartBundle immediately followed by
Nevermind, this is discussed on the PR linked.
On Mon, Oct 21, 2019 at 2:11 PM Kyle Weaver wrote:
> Do you know why an empty bundle might be created?
>
> On Mon, Oct 21, 2019 at 1:42 PM Jan Lukavský wrote:
>
>> Hi,
>>
>> when debugging a flaky ParDoLifecycleTest in FlinkRunner, I have found a
High Priority Dependency Updates Of Beam Python SDK:
Dependency Name
Current Version
Latest Version
Release Date Of the Current Used Version
Release Date Of The Latest Release
JIRA Issue
mock
2.0.0
3.0.5
2019-05-20
Hi,
when debugging a flaky ParDoLifecycleTest in FlinkRunner, I have found a
situation, where Flink might create empty bundle - i.e. call
@StartBundle immediately followed by @FinishBundle, with no elements
inside the bundle. That is what breaks the ParDoLifecycleTest, because
the test
Hi,
I found that in `SdkHarness` do not stop the `SdkWorker` when finish. We
should add the logic for stop the `SdkWorker` in `SdkHarness`. More detail
can be found [1].
There are two approaches to solve this issue:
Approach 1: We can add a Fn API for teardown purpose and the runner will
25 matches
Mail list logo
