Hi everybody I am working with Karthi on this connector. I think I can address some of the concerns raised by Danny et al. I do not have write access to the FLIP yet, but I will get it and do some edits. Also regarding the motivation, as suggested by Martijn.
Q1) The fact we are using the remote write feature is not covered beyond the code example. Can we add details on this to make it clear? Additionally would this support _any_ Prometheus server or do we need to enable remote endpoint feature on the server? A1) We use the remote-write API. The server must provide a standard remote-write endpoint. Remote-write specs do not say anything about authentication. At the moment we are planning to support 1/ unauthenticated requests, 2/ AWS signed requests for AMP. The idea is the signer interface allows transformations of the request URL. Request payload cannot be modified and must be Protobuf, as by spec. Q2) Are there any concerns around Prometheus versioning or is the API backwards compatible? Which versions of Prometheus will we be supporting A2) We are using the only version of Prometheus Remote-Write specs available v1.0, defined in Remote-Write spec document [1] published April 2023. There was a previous v0.1 draft version of the same specs. We will probably also be compatible with the draft version, but I still have to check the differences. Q3) With regard to the "AmazonPrometheusRequestSigner" the example has static creds. Can we integrate with the AWS Util to support all credential providers, static and dynamic? A3) Static credentials are just for the sake of the example. The current prototype implementation already uses DefaultAWSCredentialsProviderChain that supports static and dynamic credentials. We can make the credential provider configurable. Lorenzo [1] https://prometheus.io/docs/concepts/remote_write_spec/ [2] https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit On Wed, 24 May 2023 at 19:37, Nicora, Lorenzo <nicor...@amazon.co.uk> wrote: > *From: *Martijn Visser <martijnvis...@apache.org> > > *Subject: RE: [EXTERNAL][DISCUSS] FLIP-229: Prometheus Sink Connector* > > *Date: *22 May 2023 at 14:49:20 BST > > *To: *<dev@flink.apache.org> > > *Reply-To: *<dev@flink.apache.org> > > > > Hi all, > > > For example, a user might want to read in logs, perform some aggregations > > and publish it into a metrics store for visualisation. This might be a > great use-case for reducing the cardinality of metrics! > > I can see that. What I would like to see in the FLIP is a proposal on the > boundaries of the metrics reporter vs the Prometheus sink. I think it's > important that we make clear when to use a metric reporter and when not. I > can imagine that there will be Flink users who think that they can get data > from the metric reporter, make aggregrations in Flink and then store it > using the Prometheus sink. > > Overall, I think more context must be added to the FLIP, especially on the > motivation. > > Best regards, > > Martijn > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan <kar...@karthitect.com> > wrote: > > > Hi Lijie > > Thank you for pointing this out. I've corrected it [1]. Also, this page > [2] still shows 178 and 229 as available, which is why I picked it up. > > Thanks > Karthi > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > [2] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > On May 15, 2023, at 9:37 PM, Lijie Wang <wangdachui9...@gmail.com> wrote: > > > Hi Karthi, > > I think you are using a wrong FLIP id, the FLIP-229 has already be used[1]. > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > Best, > Lijie > > Martijn Visser <martijnvis...@apache.org> 于2023年5月16日周二 04:44写道： > > Hi Karthi, > > Thanks for the FLIP and opening up the discussion. My main question is: why > should we create a separate connector and not use and/or improve the > existing integrations with Prometheus? I would like to understand more so > that it can be added to the motivation of the FLIP. > > Best regards, > > Martijn > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan <kar...@karthitect.com> > wrote: > > > Hello all, > > We would like to start a discussion thread on FLIP-229: Prometheus Sink > Connector [1] where we propose to provide a sink connector for Prometheus > [2] based on the Async Sink [3]. Looking forward to comments and > > feedback. > > Thank you. > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > [2] https://prometheus.io/ > [3] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 > Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship > Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, > UK Establishment No. BR019315 > > >
