Hi all,
Thanks for your confirmation Robert! :)
Thanks for share the details about the state discussion Luke! :)
The MapState is a bit complex, I think it's better to add some detail
design doc when we deal with the map state supported.
I will create JIRAs and follow up on subsequent
I wanted to add some more details about the state discussion.
BEAM-7000 is about adding support for a gRPC message saying that the SDK is
now blocked on one of its requests. This would allow for an easy
optimization on the runner side where it gathers requests and is able to
batch them knowing
The list looks good to me. Thanks for summarizing. Feel free to dive
into any of these issues yourself :).
On Fri, Aug 2, 2019 at 6:24 PM jincheng sun wrote:
>
> Hi all,
>
>
> Thanks a lot for sharing your thoughts!
>
>
> It seems that we have already reached consensus for the following items.
Hi all,
Thanks a lot for sharing your thoughts!
It seems that we have already reached consensus for the following items.
Could you please read through them again and double-check if you all agree
with these? If yes, then I would start creating JIRA issues for those that
don’t yet have a JIRA
Thanks for share the detail of the current StandardCoders Max!
That's true, Flink may should defined some of coders, And I will share the
POC in the Flink Python UDFs DISCUSS Thread later :)
Best,
Jincheng
Maximilian Michels 于2019年7月31日周三 下午2:53写道：
> Hi Jincheng,
>
> Thanks for getting back to
Hi Jincheng,
Thanks for getting back to us.
> For the next major release of Flink, we plan to add Python user defined
> functions(UDF, UDTF, UDAF) support in Flink and I have go over the Beam
> portability framework and think that it is perfect for our requirements.
> However we also find
Yep, Python support under active development, e.g.
https://github.com/apache/beam/pull/9188
On Wed, Jul 31, 2019 at 9:24 AM jincheng sun
wrote:
> Thanks a lot for sharing the link. I take a quick look at the design and
> the implementation in Java and think it could address my concern. It seems
Thanks a lot for sharing the link. I take a quick look at the design and
the implementation in Java and think it could address my concern. It seems
that it's still not supported in the Python SDK Harness. Is there any plan
on that?
Robert Bradshaw 于2019年7月30日周二 下午12:33写道：
> On Tue, Jul 30, 2019
On Tue, Jul 30, 2019 at 11:52 AM jincheng sun
wrote:
>
>>> Is it possible to add an interface such as `isSelfContained()` to the
>>> `Coder`? This interface indicates
>>> whether the serialized bytes are self contained. If it returns true,
>>> then there is no need to add a prefixing length.
>>>
On Mon, Jul 29, 2019 at 4:14 PM jincheng sun
wrote:
> Hi Robert,
>
> Thanks for your detail comments, I would have added a few pointers inline.
>
> Best,
> Jincheng
>
> Robert Bradshaw 于2019年7月29日周一 下午12:35写道：
>
>> On Sun, Jul 28, 2019 at 6:51 AM jincheng sun
>> wrote:
>> >
>> > Hi, Thomas &
Hi Robert,
Thanks for your detail comments, I would have added a few pointers inline.
Best,
Jincheng
Robert Bradshaw 于2019年7月29日周一 下午12:35写道：
> On Sun, Jul 28, 2019 at 6:51 AM jincheng sun
> wrote:
> >
> > Hi, Thomas & Robert, Thanks for your comments and providing relevant
> discussions and
On Sun, Jul 28, 2019 at 6:51 AM jincheng sun wrote:
>
> Hi, Thomas & Robert, Thanks for your comments and providing relevant
> discussions and JIRA links, very helpful to me!
>
> I am glad to see your affirmative response, And I am glad to add my thoughts
> on the comment you left:
>
Hi, Thomas & Robert, Thanks for your comments and providing relevant
discussions and JIRA links, very helpful to me!
I am glad to see your affirmative response, And I am glad to add my
thoughts on the comment you left:
-
>> There are two distinct levels at which one can talk
On Thu, Jul 25, 2019 at 5:31 AM Thomas Weise wrote:
>
> Hi Jincheng,
>
> It is very exciting to see this follow-up, that you have done your research
> on the current state and that there is the intention to join forces on the
> portability effort!
>
> I have added a few pointers inline.
>
>
Hi Jincheng,
It is very exciting to see this follow-up, that you have done your research
on the current state and that there is the intention to join forces on the
portability effort!
I have added a few pointers inline.
Several of the issues you identified affect our usage of Beam as well.
Hi all,
Thanks Max and all of your kind words. :)
Sorry for the late reply as I'm busy working on the Flink 1.9 release. For
the next major release of Flink, we plan to add Python user defined
functions(UDF, UDTF, UDAF) support in Flink and I have go over the Beam
portability framework and think
Fully agree that this is an effort that goes beyond changing a type
parameter but I think we have a chance here to cooperate between the two
projects. I would be happy to help out where I can.
I'm not sure at this point what exactly is feasible for reuse but I
would imagine the Runner-related
Hi Kenn, I think you are right, the Python SDK harness can be shared to
Flink, and also need to add some new primitive operations. Regarding
runner-side, I think most of the code which in runners/java-fun- Execution
is can be shared(but need some improvement, such as FnDataService), some of
them
It seems to me that the most valuable code to share and keep up with is the
Python/Go/etc SDK harness; they would need to be enhanced with new
primitive operations. So you would want to depend directly and share the
original proto-generated classes too, which Beam publishes as separate
artifacts
Hi Jincheng,
Copying code is a solution for the short term. In the long run I'd like
the Fn services to be a library not only for the Beam portability layer
but also for other projects which want to leverage it. We should thus
make an effort to make it more generic/extensible where necessary
Hi Reuven,
I think you have provided an optional solution for other community which
wants to take advantage of Beam's existing achievements. Thank you very
much!
I think the Flink community can choose to copy from Beam's code or choose
to rely directly on the beam's class library. The Flink
Makes sense to me. I don't see any problem.
Kenn
On Mon, Apr 22, 2019 at 12:08 AM jincheng sun
wrote:
> Hi Kenn,
>
> Thanks for your reply, and explained the design of WindowValue clearly!
>
> At present, the definitions of `FnDataService` and `BeamFnDataClient` in
> Data Plane are very clear
Hi Kenn,
Thanks for your reply, and explained the design of WindowValue clearly!
At present, the definitions of `FnDataService` and `BeamFnDataClient` in
Data Plane are very clear and universal, such as: send(...)/receive(...).
If it is only applied in the project of Beam, it is already very
WindowedValue has always been an interface, not a concrete representation:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
Thank you! And have a nice weekend!
Lukasz Cwik 于2019年4月20日周六 上午1:14写道：
> I have added you as a contributor.
>
> On Fri, Apr 19, 2019 at 9:56 AM jincheng sun
> wrote:
>
>> Hi Lukasz,
>>
>> Thanks for your affirmation and provide more contextual information. :)
>>
>> Would you please give me
I have added you as a contributor.
On Fri, Apr 19, 2019 at 9:56 AM jincheng sun
wrote:
> Hi Lukasz,
>
> Thanks for your affirmation and provide more contextual information. :)
>
> Would you please give me the contributor permission? My JIRA ID is
> sunjincheng121.
>
> I would like to
Since I don't think this is a contentious change.
On Fri, Apr 19, 2019 at 9:25 AM Lukasz Cwik wrote:
> Yes, using T makes sense.
>
> The WindowedValue was meant to be a context object in the SDK harness that
> propagates various information about the current element. We have discussed
> in the
Yes, using T makes sense.
The WindowedValue was meant to be a context object in the SDK harness that
propagates various information about the current element. We have discussed
in the past about:
* making optimizations which would pass around less of the context
information if we know that the
Hi Beam devs,
I read some of the docs about `Communicating over the Fn API` in Beam. I
feel that Beam has a very good design for Control Plane/Data Plane/State
Plane/Logging services, and it is described in document. When communicating between Runner and SDK Harness, the
DataPlane API will be
29 matches
Mail list logo
