Hi,
Just wanted to mention that I updated this document with one detail
https://s.apache.org/beam-gcp-debuggability
Date
Changes
Sept 8, 2020
-
Clarified that the InstructionRequest/Control Channel will be used in
“Proposal: SDKHs to Report non-bundle metrics.”
May 15, 2020
-
egner wrote the metrics for Java.
>>> >
>>> > Best
>>> > -P.
>>> >
>>> > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> I have created a propo
Thanks everyone. I was able to collect a lot of good feedback from everyone
who contributed. I am going to wrap it up for now and label the design as
"Design Finalized (Unimplemented)".
I really believe we have made a much better design than I initially wrote
up. I couldn't have done it without th
Thanks to all who have spent their time on this, there were many great
suggestions, just another reminder that tomorrow I will be finalizing the
documents, unless there are any major objections left. Please take a look
at it if you are interested.
I will still welcome feedback at any time :).
But
Thanks again for more feedback :). I have iterated on things again. I'll
report back at the end of the week. If there are no major disagreements
still, I'll close the discussion, believe it to be in a good enough state
to start some implementation. But welcome feedback.
Latest changes are changing
Thanks for the great feedback so far :). I've included many new ideas, and
made some revisions. Both docs have changed a fair bit since the initial
mail out.
https://s.apache.org/beam-gcp-debuggability
https://s.apache.org/beam-histogram-metrics
PTAL and let me know what you think, and hopefully
Thanks everyone so far for taking a look so far :).
I am hoping to have this finalize the two reviews by the end of next week,
May 15th.
I'll continue to follow up on feedback and make changes, and I will add
some more mentions to the documents to draw attention
https://s.apache.org/beam-gcp-deb
Thanks, also took a look and left some comments.
On Tue, May 5, 2020 at 6:24 PM Alex Amato wrote:
> Hello,
>
> I created another design document. This time for GCP IO Debuggability
> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
> This is for monitoring request coun
>
>> > Best
>> > -P.
>> >
>> > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote:
>> >>
>> >> Hello,
>> >>
>> >> I have created a proposal for Apache Beam FN API to support Histogram
>> Style Metrics. Which d
Hello,
I created another design document. This time for GCP IO Debuggability
Metrics. Which defines some new metrics to collect in the GCP IO libraries.
This is for monitoring request counts and request latencies.
Please take a look and let me know what you think:
https://s.apache.org/beam-gcp-de
p getting it merged, but it's worth looking at the work. I also
> remember Scott Wegner wrote the metrics for Java.
> >
> > Best
> > -P.
> >
> > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote:
> >>
> >> Hello,
> >>
> >&
>
> Best
> -P.
>
> On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote:
>>
>> Hello,
>>
>> I have created a proposal for Apache Beam FN API to support Histogram Style
>> Metrics. Which defines a method to collect Histogram style metrics and pass
>>
Mon, May 4, 2020 at 3:33 PM Alex Amato wrote:
> Hello,
>
> I have created a proposal for Apache Beam FN API to support Histogram
> Style Metrics
> <https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit#>.
> Which defines a method to collect His
Hello,
I have created a proposal for Apache Beam FN API to support Histogram Style
Metrics
<https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit#>.
Which defines a method to collect Histogram style metrics and pass them
over the FN API.
I would love to hea
rote:
>
>> Hello,
>>
>> I have created a proposal for Apache Beam FN API to support Histogram
>> Style Metrics
>> <https://docs.google.com/document/d/1MtBZYV7NAcfbwyy9Op8STeFNBxtljxgy69FkHMvhTMA/edit#heading=h.c6fjf0g6rsbc>.
>> Which defines a method to c
Hi Alex!
Thanks for the proposal. I've created
https://s.apache.org/beam-histogram-metrics
On Mon, May 4, 2020 at 2:44 PM Alex Amato wrote:
> Hello,
>
> I have created a proposal for Apache Beam FN API to support Histogram
> Style Metrics
> <https://doc
Hello,
I have created a proposal for Apache Beam FN API to support Histogram Style
Metrics
<https://docs.google.com/document/d/1MtBZYV7NAcfbwyy9Op8STeFNBxtljxgy69FkHMvhTMA/edit#heading=h.c6fjf0g6rsbc>.
Which defines a method to collect Histogram style metrics and pass them
over the FN API.
Hello,
I have rewritten most of the proposal. Though I think that there is some
more research that needs to be done to get the Metric specification
perfect. I plan to do more research, and would like to ask you all for more
help to make this proposal better. In particular, now that the metrics
for
That sounds like a very reasonable choice -- given the discussion seemed to
be focusing on the differences between these two categories, separating
them will allow the proposal (and implementation) to address each category
in the best way possible without needing to make compromises.
Looking forwa
Hello,
I just wanted to give an update .
After some discussion, I've realized that its best to break up the two
concepts, with two separate way of reporting monitoring data. These two
categories are:
1. Metrics - Counters, Gauges, Distributions. These are well defined
concepts for monitori
I agree that the user/system dichotomy is false, the real question of how
counters can be scoped to avoid accidental (or even intentional)
interference. A system that entirely controls the interaction between the
"user" (from its perspective) and the underlying system can do this by
prefixing all r
One reason I resist the user/system distinction is that Beam is a
multi-party system with at least SDK, runner, and pipeline. Often there may
be a DSL like SQL or Scio, or similarly someone may be building a platform
for their company where there is no user authoring the pipeline. Should
Scio, SQL,
On Fri, Apr 13, 2018 at 5:00 PM Robert Bradshaw wrote:
> On Fri, Apr 13, 2018 at 3:28 PM Andrea Foegler wrote:
>
>> Thanks, Robert!
>>
>> I think my lack of clarity is around the MetricSpec. Maybe what's in my
>> head and what's being proposed are the same thing. When I read that the
>> Metric
On Fri, Apr 13, 2018 at 4:30 PM Alex Amato wrote:
> There are a few more confusing concepts in this thread
> *Name*
>
>- Name can mean a *"string name"* used to refer to a metric in a
>metrics system such as stackdriver, i.e. "ElementCount", "ExecutionTime"
>- Name can mean a set of *
On Fri, Apr 13, 2018 at 3:28 PM Andrea Foegler wrote:
> Thanks, Robert!
>
> I think my lack of clarity is around the MetricSpec. Maybe what's in my
> head and what's being proposed are the same thing. When I read that the
> MetricSpec describes the proto structure, that sound kind of complicate
; like we are
>>>>>>>>>>>>>>> doing user metrics right.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Ben
>>>>>>>>>>>>>>>
>>>&
>>> rober...@google.com> a écrit :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By "type" of metric, I mean both the data types (including
>>>>>>>&
t; of metric, I mean both the data types (including
>>>>>>>>>>>>>>>> their encoding) and accumulator strategy. So sumint would be a
>>>>>>>>>>>>>>>> type, as
>>>>>>
tor strategy? Specifically, what is the
>>>>>>>>>>>>>>>> "type" of sumint,
>>>>>>>>>>>>>>>> sumlong, meanlong, etc?
>>>>>>>>>>>>>>
gt;>>>>>>>>>>>>>> never get to). What I'm suggesting is that we support custom
>>>>>>>>>>>>>>>> metrics of
>>>>>>>>>>>>>>>> standard type.
>>>>>>>>>>>
prevent user defined metric
>>>>>>>>>>>>>>>> types based on the fact they just weren't used enough to
>>>>>>>>>>>>>>>> justify support.
>>>>>>>>>>>>>>>>
>>>>>>
>>>>>>>>>>> system metrivs?
>>>>>>>>>>>>>>> On Wed, Apr 11, 2018, 5:43 PM Robert Bradshaw <
>>>>>>>>>>>>>>> rober...@google.com> wrote:
>>>>>>>>>>&
>>>>>>>>>>> On Wed, Apr 11, 2018, 5:43 PM Robert Bradshaw <
>>>>>>>>>>>>>>> rober...@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>
nd custom metric types. I would
>>>>>>>>>>>>>>> propose
>>>>>>>>>>>>>>> the MetricSpec field be augmented with an additional field
>>>>>>>>>>>>>>> "type"
aggregation). Summing or maxing
>>>>>>>>>>>>>> over ints
>>>>>>>>>>>>>> would be a typical example. Though we could pursue making this
>>>>>>>>>>>>>> opaque to
>>&
that it did not itself
>>>>>>>>>>>>> understand the
>>>>>>>>>>>>> semantic meaning of. (It would probably simplify much of the
>>>>>>>>>>>>> specialization
>>>>>>>
or every
>>>>>>>>>>>> type X one would have a single URN for UserMetric and it spec would
>>>>>>>>>>>> designate the type and payload designate the (qualified) name.
>>>>>>>>>>>>
>>>
designate
>>>>>>>>>>> the type and payload designate the (qualified) name.
>>>>>>>>>>>
>>>>>>>>>>> - Robert
>>>>>>>>>>>
>>>>>>>>>>>
>
;>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 5:12 PM Alex Amato
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>&g
>>
>>>>>>>>> On Wed, Apr 11, 2018 at 5:12 PM Alex Amato
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thank you everyone for your feedback so far.
>>>>>>>>>> I have made a
entity, so I have restructured some of the protos a little
>>>>>>>>> bit.
>>>>>>>>>
>>>>>>>>> The point of this change was to futureproof the possibility of
>>>>>>>>> allowin
change was to futureproof the possibility of
>>>>>>>> allowing custom user metrics, with custom aggregation functions for its
>>>>>>>> metric updates.
>>>>>>>> Now that each metric has an aggregation_entity asso
>>>>>> the opaque bytes metric updates, without deserializing them. These are
>>>>>>> forwarded to user provided code which then would deserialize the metric
>>>>>>> update payloads and perform the custom aggregations.
>>>>>
;>>>> update payloads and perform the custom aggregations.
>>>>>>
>>>>>> I think it has also simplified some of the URN metric protos, as they
>>>>>> do not need to keep track of ptransform names inside themselves now. The
>>
;>>> result is simpler structures, for the metrics as the entities are pulled
>>>>> outside of the metric.
>>>>>
>>>>> I have mentioned this in the doc now, and wanted to draw attention to
>>>>> this particular revision.
>>&
oned this in the doc now, and wanted to draw attention to
>>>> this particular revision.
>>>>
>>>>
>>>>
>>>> On Tue, Apr 10, 2018 at 9:53 AM Alex Amato wrote:
>>>>
>>>>> I've gathered a lot of feedback so f
10, 2018 at 9:53 AM Alex Amato wrote:
>>>
>>>> I've gathered a lot of feedback so far and want to make a decision by
>>>> Friday, and begin working on related PRs next week.
>>>>
>>>> Please make sure that you provide your feedback b
>
>>> Please make sure that you provide your feedback before then and I will
>>> post the final decisions made to this thread Friday afternoon.
>>>
>>>
>>> On Thu, Apr 5, 2018 at 1:38 AM Ismaël Mejía wrote:
>>>
>>>> Nice, I crea
g on related PRs next week.
>>
>> Please make sure that you provide your feedback before then and I will
>> post the final decisions made to this thread Friday afternoon.
>>
>>
>> On Thu, Apr 5, 2018 at 1:38 AM Ismaël Mejía wrote:
>>
>>> Nice, I
in
>> future discussions, website, etc.
>>
>> https://s.apache.org/beam-fn-api-metrics
>>
>> Thanks for sharing.
>>
>>
>> On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw
>> wrote:
>> > Thanks for the nice writeup. I added some c
smaël Mejía wrote:
> Nice, I created a short link so people can refer to it easily in
> future discussions, website, etc.
>
> https://s.apache.org/beam-fn-api-metrics
>
> Thanks for sharing.
>
>
> On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw
> wrote:
> >
Nice, I created a short link so people can refer to it easily in
future discussions, website, etc.
https://s.apache.org/beam-fn-api-metrics
Thanks for sharing.
On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw wrote:
> Thanks for the nice writeup. I added some comments.
>
> On Wed, Ap
Thanks for the nice writeup. I added some comments.
On Wed, Apr 4, 2018 at 1:53 PM Alex Amato wrote:
> Hello beam community,
>
> Thank you everyone for your initial feedback on this proposal so far. I
> have made some revisions based on the feedback. There were some larger
> questions asking abo
Hello beam community,
Thank you everyone for your initial feedback on this proposal so far. I
have made some revisions based on the feedback. There were some larger
questions asking about alternatives. For each of these I have added a
section tagged with [Alternatives] and discussed my recommendat
essing
>>>
>>> The document does require a strong foundation in the Apache Beam model
>>> and
>>> a good understanding of the prior shared docs:
>>> * How to process a bundle: https://s.apache.org/beam-fn-api
>>> -processing-a-bundle
>>
does require a strong foundation in the Apache Beam model and
a good understanding of the prior shared docs:
* How to process a bundle: https://s.apache.org/beam-fn-api
-processing-a-bundle
* How to send and receive data: https://s.apache.org/beam-fn-api
-send-and-receive-data
I could really use
-state-api-and-bundle-processing
>
> The document does require a strong foundation in the Apache Beam model and
> a good understanding of the prior shared docs:
> * How to process a bundle: https://s.apache.org/beam-fn-api
> -processing-a-bundle
> * How to send and receive data: https
document does require a strong foundation in the Apache Beam model and
a good understanding of the prior shared docs:
* How to process a bundle: https://s.apache.org/beam-fn-api
-processing-a-bundle
* How to send and receive data: https://s.apache.org/beam-fn-api
-send-and-receive-data
I could really use
gt; in your mail.
>
> * How to process a bundle:
> https://s.apache.org/beam-fn-api-processing-a-bundle
> * How to send and receive data:
> https://s.apache.org/beam-fn-api-send-and-receive-data
>
> By the way, is there a way to find them from the Beam website ?
>
>
> On
Thanks Lukasz. The following two links were somehow incorrectly formatted
in your mail.
* How to process a bundle:
https://s.apache.org/beam-fn-api-processing-a-bundle
* How to send and receive data:
https://s.apache.org/beam-fn-api-send-and-receive-data
By the way, is there a way to find them
share the overview:
https://s.apache.org/beam-fn-api
And for those of you who have been following this thread and contributors
focusing on Runner integration with Apache Beam:
* How to process a bundle: https://s.apache.org/beam-fn-api-processing-a-
bundle
* How to send and receive data:
> > > can
> > > > > > > deploy code/dependencies on demand).
> > > > > > > A user creates a pipeline saying which docker container they
> want
> > > to
> > > > > use
> > > > > > > (this starts
> > > > containers in a cluster manager of their choice (scaling up or
> down
> > > the
> > > > > > number of instances depending on demand/load/...).
> > > > > > A runner then interacts with the docker containers over the gRPC
> > > > s
> j...@nanthrax.net
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Luke,
> > > > >>
> > > > >> that's really great and very promising !
> > > > >>
> > > > >> It
;> using gRPC is once the docker container is running, then we can
> > > "interact"
> > > >> with the container to spread and delegate processing to the docker
> > > >> container, correct ?
> > > >> The users/devops have to setup t
t; via the pipeline options and used by gRPC ?
> > >>
> > >> Thanks Luke !
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On 01/19/2017 03:56 PM, Lukasz Cwik wrote:
> > >>
> >
nd of container registry) is
> set
> >> via the pipeline options and used by gRPC ?
> >>
> >> Thanks Luke !
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On 01/19/2017 03:56 PM, Lukasz Cwik wrote:
> >>
> >>&
>>
>> On 01/19/2017 03:56 PM, Lukasz Cwik wrote:
>>
>>> I have been prototyping several components towards the Beam technical
>>> vision of being able to execute an arbitrary language using an arbitrary
>>> runner.
>>>
>>> I wo
Cwik wrote:
>
>> I have been prototyping several components towards the Beam technical
>> vision of being able to execute an arbitrary language using an arbitrary
>> runner.
>>
>> I would like to share this overview [1] of what I have been working
>> towards. I also s
tial implementation.
1: https://s.apache.org/beam-fn-api
2: https://github.com/apache/beam/pull/1801
Please comment on the overview within this thread, and any specific code
comments on the PR directly.
Luke
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
ecute an arbitrary language using an arbitrary
>> runner.
>>
>> I would like to share this overview [1] of what I have been working
>> towards. I also share this PR [2] with a proposed API, service definitions
>> and partial implementation.
>>
>> 1: https://s
with a proposed API, service definitions
> and partial implementation.
>
> 1: https://s.apache.org/beam-fn-api
> 2: https://github.com/apache/beam/pull/1801
>
> Please comment on the overview within this thread, and any specific code
> comments on the PR directly.
>
> Luke
>
partial implementation.
1: https://s.apache.org/beam-fn-api
2: https://github.com/apache/beam/pull/1801
Please comment on the overview within this thread, and any specific code
comments on the PR directly.
Luke
73 matches
Mail list logo
