[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125179#comment-17125179 ] Badrul Chowdhury commented on IOTDB-519: Hi [~hxd], Thanks for following up on this. I started working on this last week and I am making slow progress. That said, I hope to make meaningful progress on this in June - I'll keep you posted throughout as I hit significant milestones/issues along the way. Thanks, Badrul > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124606#comment-17124606 ] Xiangdong Huang commented on IOTDB-519: --- Hi, any updates? > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102134#comment-17102134 ] Badrul Chowdhury commented on IOTDB-519: Thanks for providing an example [~hxd], it helps a lot. Yes, my proposal was not accepted by GSoC unfortunately. Regardless, I would love to work on the Prometheus integration project during the summer after I wrap up my course work for the current semester. Let me return to this after my final exams on 05/22. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101135#comment-17101135 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~badrul_c], sorry (maybe) because of something error of GSoc organization, I did not receive any GSoc proposal from Google or other organizations... So, maybe we have no opportunity to work as a GSoC task... But I still look forward to your contribution if you are interested in this issue. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099488#comment-17099488 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~badrul_c], sorry for late reply. Is there any progress? >> A.ph.B.ph.ph.C I am not entirely sure why we need a placeholder, could you share an example of its use please? Ok,... let's see the follwoing example: Suppose a user creates a time series "root.sg.China.Beijing.DC1.H1" to represent a device (here, the device is a server) where country = China, city = Beijing, data center = DC1, and model = Dell. Then we have $order(country)=1$, $order(city)=2$, $order(data center)=3$ and $order(model)=4$, right? Now if we have more timeseries which have all the above tags, everything is fine... like: root.sg.China.Shanghai.DC2.Dell, root.sg.USA.NewYork.DC1.IBM, ... Consider the user wants to create another timeseries, to collect the data of all the devices that model=Dell, then he/she may create a time series which has only one tag: model = dell. If so, how to create a time series in IoTDB? root.sg.dell? or root.sg.ph.ph.dell? It has to be the latter. It is because we only know $order(model) = 4$. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094576#comment-17094576 ] Badrul Chowdhury commented on IOTDB-519: Sounds good, that resolves any potential licensing issues re the algorithm. I traced through the algorithm and it makes sense to me, although I am not quite sure why we need the placeholder in the path that is returned by generatePath(). > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094532#comment-17094532 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~badrul_c], Welcome back. Yes, it is a private project. But I think I introduce the algorithm clear. If there is other problem, let me know. I am the proposer of the algorithm, so feel free to implement it and donate it to IoTDB project. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094526#comment-17094526 ] Badrul Chowdhury commented on IOTDB-519: Hi [~hxd], Sorry for the late reply - I was caught up in exams and assignments for the past week and a half. I found some time to review your proposal this past weekend and it looks good overall IMO. A few follow up comments/questions: >> (As the following design comes from one of our commetrial application. We >>can use it only if someone of you implement it and donate it into Apache >>IoTDB Repository. And all variants of this algorithm need to claim it is >>derived from this algorithm.) Could you give pointers to this application or is it proprietary? >> (Improvement: considering the memory cost, we can only keep a part of the >>table in memory as a cache.) I would suggest adding another config value in iotdb-engine.properties specifying the max number of entries in the in-memory Map> table, what do you think? > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png, > prometheus-iotdb-schema.md > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071320#comment-17071320 ] Badrul Chowdhury commented on IOTDB-519: Sounds good [~hxd]! I will go ahead and submit my application today. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070793#comment-17070793 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~badrul_c] , I have read your proposal. I think the proposal is fine for the application. You are right that we indeed need to make a consensus for the schema mapping method later. But it does not impact the application result, I think. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070722#comment-17070722 ] Badrul Chowdhury commented on IOTDB-519: [~hxd] [~VenkatDatta] I added a new section with a proposal for a new approach for handling out-of-order labels, PTAL when you get the chance: [Filtering or Aggregating on Out-of-order Labels|https://docs.google.com/document/d/1UYOUd0YomS6NUitG9Ko0gqzYapn6eqhFuy_5F3NKVxg/edit#heading=h.fpeu0un0czfe] Thanks! Badrul > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069238#comment-17069238 ] Badrul Chowdhury commented on IOTDB-519: [~VenkatDatta] I reviewed your proposal - in general, it looks good. My only concern is mapping a single metric to a storage group. We cannot control the number of metrics emitted by an app (and there is no upper limit), so having a single storage group per metric may be very inefficient. Please also feel free to review my proposal and leave your comments. We will be implementing this together so the more aligned we are, the better :) > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069236#comment-17069236 ] Badrul Chowdhury commented on IOTDB-519: [~hxd] Thanks for the feedback! I have gone over the past discussions regarding the mapping and added 2 sections to my proposal summarizing my thoughts: [Converting Data from Prometheus to IoTDB format|https://docs.google.com/document/d/1UYOUd0YomS6NUitG9Ko0gqzYapn6eqhFuy_5F3NKVxg/edit#heading=h.8c0k9vrknfwy] [Filter|https://docs.google.com/document/d/1UYOUd0YomS6NUitG9Ko0gqzYapn6eqhFuy_5F3NKVxg/edit#heading=h.f071o1u7ptz8] [Aggregations|https://docs.google.com/document/d/1UYOUd0YomS6NUitG9Ko0gqzYapn6eqhFuy_5F3NKVxg/edit#heading=h.lp3ayq1dx1y1] [How Many Storage Groups?|https://docs.google.com/document/d/1UYOUd0YomS6NUitG9Ko0gqzYapn6eqhFuy_5F3NKVxg/edit#heading=h.pbiewnx0k895] > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068918#comment-17068918 ] Badrul Chowdhury commented on IOTDB-519: [~VenkatDatta], thanks for sharing, I will review your proposal today. My email address is [badrulchowdhur...@gmail.com.|mailto:badrulchowdhur...@gmail.com.] [~hxd], thanks for the feedback. Let me take another look and modify the mapping accordingly. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068878#comment-17068878 ] Venkat Datta commented on IOTDB-519: Hi [~hxd], Thanks a lot for letting me know what to prioritize upon. I will keep brainstorming and keep you guys posted with any new ideas. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068826#comment-17068826 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~VenkatDatta], Using Antlr4 to add new SQL syntax is not very difficult, you can design the syntax first if you want. But for me, I'd like to define the schema mapping first because if we can overcome the issue, then this task has no other failure risk... Actually I led a guy finishing the schema mapping between Kairos and IOTDB last year (in a private project for some real-world application). I did not tell you the details now because # I do not want to limit your minds; # I do not think the solution is the best and I hope you may provide a better solution. So, I think all of us can take some time to consider more. If we still have no clear idea, I can share what we did, and then let's consider how to implement and improve it. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068559#comment-17068559 ] Venkat Datta commented on IOTDB-519: Hi [~hxd], I have given the necessary permission for the doc. What should be our next steps? I was thinking of exploring the Antlr4 and see the feasibility of adding support of labels in our SQL syntax. Also, should I dive deep upon the way to store the labels metadata in our IoTDB current architecture and come up with a detailed design? Please let me know. Thanks, Venkat > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068448#comment-17068448 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~badrul_c] , I have read your proposal. It is good. But please pay more attention to the mapping between Prometheus schema and IoTDB schema. I think the comments in this issue can help you know why we care about it. Hi [~VenkatDatta], my email is sainthxd (gmail) Best, > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068272#comment-17068272 ] Venkat Datta commented on IOTDB-519: Hi [~hxd], [~badrul_c] I have consolidated both the documents into a single doc. There are still missing pieces that need to be addressed. Please have a look at it and let me know. Also, Can you guys share the mail ids for which I can give permissions to collaborate on the doc? Doc: https://docs.google.com/document/d/1DZL1L0GccLvHbMJyop62cbaSvu1tkJ6Pp1Fp6FsR09U Thanks, Venkat > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066597#comment-17066597 ] Venkat Datta commented on IOTDB-519: Hi [~badrul_c], I'm happy to collaborate with you on building the Prometheus connector. I have interpreted my views in the above comments. I'm currently working on consolidating everything into a single document and share it for a review. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066450#comment-17066450 ] Badrul Chowdhury commented on IOTDB-519: Hi [~VenkatDatta], I was hoping to work on this for GSoC 2020, here is my proposal: [https://docs.google.com/document/d/1UYOUd0YomS6NUitG9Ko0gqzYapn6eqhFuy_5F3NKVxg/edit?usp=sharing.] I'd be happy to work with you on implementing the connector for Prometheus. If you would rather work on this by yourself, I can pivot to a different connector too. Either way, please let me know! Thanks, Badrul > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064827#comment-17064827 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~VenkatDatta], It is a pity that you cannot join GSoc. But indeed it does not impact you continue to join this task :D . So... welcome to the community! Yes if we have a beautiful implementation ( and some applications if possible), it is a good time to share the experience to the PromCon. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064826#comment-17064826 ] Venkat Datta commented on IOTDB-519: Sure, [~hxd]. I will capture all the discussion and designs into a single document and share it with you. Also, I'm not eligible for the GSoC (3-month program). I want to work upon this as an individual contributor. Please do let me know if you want me to explore the Antlr4 for coming up with newer SQL syntax. I also feel that will be more beneficial for the project. I was also looking out at previous conferences of PromCon, where people shared experiences on the Prometheus integrations. I think we should aim for [this year|https://promcon.io/2020-vancouver]. Thanks, Venkat > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064228#comment-17064228 ] Xiangdong Huang commented on IOTDB-519: --- Hi [~VenkatDatta], looks good! Ok, let's suppose we have a middleware (o a new API module of IoTDB) for label-based data ingestion and query. For example, Insert a data point: 'insert(Map timeseries, timestamp, data_value)', or, Insert a row record that belong to the same device: insert(Map device, timestamp, List measurements, List data_values), or, Insert a batch of points: 'insert(Map timeseries, List timestamps, List values)'. (These are just for examples, you do not need to implement them all). And, Query raw data point: "query(Map some_timeseries, List, time range, limit)" (Query is very diverse, so... just define some queries that Prometheus may use). Then, you just need to have an algorithm to decide how to map the label/values to a timeseries or some timeseries with "*" (Remember that if a series is "root.a.b.c", then "root.a.*.c" can match it while "root.a.b.*.c" can not match it.). Then, translate the label/values into timeseries paths and call IoTDB's session API or SQL to run the command. It is implementation-free, no matter whether you build an invert index or put the label name into "IOTDB path", right? I think we can confirm the design of these interfaces in your proposal. Then we still have time to decide how to implement the feature. A more beautiful API design (for queries) is we define new SQLsyntax for IoTDB to support the labels. However, it requires you know Antlr4. I am not sure whether we have enough time in 3 months to do that. Then, let's discuss how to implement the two approaches. The first approach, you may still need to persist some content. For example, when users query "where region = A and host =B", you do not know it is "root.*.region.A.*.host.B" or "root.*.region.A.host.B". Actually, if you have persist some content about the mapping, then maybe you do not need to put the label name into the path any more. Finally, I suggest putting all designs or discussion documents into one doc. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-519) Apache IoTDB integration with Prometheus
[ https://issues.apache.org/jira/browse/IOTDB-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063617#comment-17063617 ] Venkat Datta commented on IOTDB-519: Hi [~hxd], sorry for the late reply. I was going through the system design of IoTDB and the label's support in other TSDB. I went through the approaches you had suggested for schema handling. I have written a document on summarising the approaches and pros/cons of them. I feel there are a lot of dots that I might be missing. Please review and let me know. Doc: https://docs.google.com/document/d/15cWNzejaGawulBfQvNYM_lrzIChCqlFAedUIhhxxrKE Thanks a lot. > Apache IoTDB integration with Prometheus > > > Key: IOTDB-519 > URL: https://issues.apache.org/jira/browse/IOTDB-519 > Project: Apache IoTDB > Issue Type: Wish > Components: Others >Reporter: Xiangdong Huang >Priority: Major > Labels: gsoc2020, iotdb, mentor > Attachments: image-2020-03-07-15-24-34-192.png, > image-2020-03-07-15-24-51-531.png, image-2020-03-07-15-25-08-037.png > > > IoTDB is a highly efficient time series database. > Prometheus is a monitoring and alerting toolkit, which supports collecting > data from other systems, servers, and IoT devices, saving data into a DB, > visualizing data and provides some query APIs. > > Prometheus allows users to use their database rather than just Prometheus DB > for storing time series databases. > This proposal is for integrating IoTDB with Prometheus. > > You should know: > * How to use Prometheus > * How to use IoTDB > * Java and Go language > difficulty: Major > mentors: > h...@apache.org -- This message was sent by Atlassian Jira (v8.3.4#803005)
