Hello I am also re-sending an old answer I sent on May 24th, that, for some reason, did not appear in the thread. ---------------------- Q1) The fact we are using the remote write feature is not covered beyond the code example. Can we add details on this to make it clear? Additionally would this support _any_ Prometheus server or do we need to enable remote endpoint feature on the server?
A1) We use the remote-write API. The server must provide a standard remote-write endpoint. Remote-write specs do not say anything about authentication. At the moment we are planning to support 1/ unauthenticated requests, 2/ AWS signed requests for AMP. The idea is the signer interface allows transformations of the request URL. Request payload cannot be modified and must be Protobuf, as by spec. Q2) Are there any concerns around Prometheus versioning or is the API backwards compatible? Which versions of Prometheus will we be supporting A2) We are using the only version of Prometheus Remote-Write specs available v1.0, defined in Remote-Write spec document [1] published April 2023. There was a previous v0.1 draft version of the same specs. We will probably also be compatible with the draft version, but I still have to check the differences. Q3) With regard to the "AmazonPrometheusRequestSigner" the example has static creds. Can we integrate with the AWS Util to support all credential providers, static and dynamic? A3) Static credentials are just for the sake of the example. The current prototype implementation already uses DefaultAWSCredentialsProviderChain that supports static and dynamic credentials. We can make the credential provider configurable. Lorenzo [1] https://prometheus.io/docs/concepts/remote_write_spec/ [2] https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit Lorenzo On Sun, 17 Sept 2023 at 09:51, Ahmed Hamdy <hamdy10...@gmail.com> wrote: > Thanks Lorenzo, > Looking forward to the PRs. > Best Regards > Ahmed Hamdy > > > On Sat, 16 Sept 2023 at 06:27, Lorenzo Nicora <lorenzo.nic...@gmail.com> > wrote: > > > Hello > > > > (apologies if this is a duplicate reply) > > > > I was working with Karthi on this connector, and I have taken over the > > development. > > We have a working version we would like to submit to the community. > > > > The renumbered FLIP-312 is also updated with more details [1]. > > Happy to answer any questions. > > > > Regards > > Lorenzo > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > > > On Mon, 21 Aug 2023, 13:06 Ahmed Hamdy, <hamdy10...@gmail.com> wrote: > > > > > Hello Karthi > > > Is this FLIP still in progress? I see the FLIP not updated & couldn't > > find > > > open JIRAs. > > > I am happy to take over if you are no longer working on this. > > > Best Regards > > > Ahmed Hamdy > > > > > > > > > On Mon, 22 May 2023 at 14:49, Martijn Visser <martijnvis...@apache.org > > > > > wrote: > > > > > > > Hi all, > > > > > > > > > For example, a user might want to read in logs, perform some > > > aggregations > > > > and publish it into a metrics store for visualisation. This might be > a > > > > great use-case for reducing the cardinality of metrics! > > > > > > > > I can see that. What I would like to see in the FLIP is a proposal on > > the > > > > boundaries of the metrics reporter vs the Prometheus sink. I think > it's > > > > important that we make clear when to use a metric reporter and when > > not. > > > I > > > > can imagine that there will be Flink users who think that they can > get > > > data > > > > from the metric reporter, make aggregrations in Flink and then store > it > > > > using the Prometheus sink. > > > > > > > > Overall, I think more context must be added to the FLIP, especially > on > > > the > > > > motivation. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan < > > > kar...@karthitect.com> > > > > wrote: > > > > > > > > > Hi Lijie > > > > > > > > > > Thank you for pointing this out. I've corrected it [1]. Also, this > > page > > > > > [2] still shows 178 and 229 as available, which is why I picked it > > up. > > > > > > > > > > Thanks > > > > > Karthi > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > > > > [2] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > > > > > On May 15, 2023, at 9:37 PM, Lijie Wang <wangdachui9...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > Hi Karthi, > > > > > > > > > > I think you are using a wrong FLIP id, the FLIP-229 has already be > > > > used[1]. > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > > > > > > > > > Best, > > > > > Lijie > > > > > > > > > > Martijn Visser <martijnvis...@apache.org> 于2023年5月16日周二 04:44写道： > > > > > > > > > > Hi Karthi, > > > > > > > > > > Thanks for the FLIP and opening up the discussion. My main question > > is: > > > > why > > > > > should we create a separate connector and not use and/or improve > the > > > > > existing integrations with Prometheus? I would like to understand > > more > > > so > > > > > that it can be added to the motivation of the FLIP. > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan < > > > > kar...@karthitect.com> > > > > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > We would like to start a discussion thread on FLIP-229: > Prometheus > > > Sink > > > > > > Connector [1] where we propose to provide a sink connector for > > > > Prometheus > > > > > > [2] based on the Async Sink [3]. Looking forward to comments and > > > > > feedback. > > > > > > Thank you. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > > > > > [2] https://prometheus.io/ > > > > > > [3] > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
