Hi VGalaxies, Thank you so much for introducing the new data subscription client feature to the Apache IoTDB community. Your detailed explanation provides a clear understanding of how this feature enhances IoTDB’s capabilities, especially in terms of real-time data processing and system integration efficiency.
Looking forward to more updates and engaging with this new feature! :) Best regrads, Sho 2024年4月8日 11:06，VGalaxies <vgalax...@apache.org> 写道： Hello everyone, I am VGalaxies, a new contributor to Apache IoTDB. I am excited to share with you a new feature that I have been working on for the past few months. The data subscription client is a new way to access data within IoTDB, distinct from the traditional method of querying data using SQL-like syntax. In scenarios where real-time data, quick response to data changes, and building highly event-driven systems are required, data subscription has greater advantages over data querying. For example, in the following two scenarios: 1. Replace extensive polling queries for large amounts of data: Avoid significant impacts on the performance of existing systems when querying frequently or when there are many data points. Also, avoid problems with determining the query scope and ensure downstream receives accurate full data. 2. Facilitate downstream system integration: It's easier to integrate with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc. There's no need to customize the logic of IoTDB's data change capture for each big data component separately, simplifying integration component design and making it easier for users. The IoTDB subscription client references some features defined by some message queue products like Kafka. It consists of 3 core concepts: Topic, Consumer, and Consumer Group. - Topic is a logical concept used by the IoTDB subscription client to classify data, serving as a channel for data publication. Producers publish data to specific topics, while consumers subscribe to these topics to receive related data. In the IoTDB subscription client, topics describe the sequence characteristics, time characteristics, presentation forms, and optional custom processing logic of subscribed data. - Consumer is an application or service in the IoTDB subscription client responsible for receiving and processing data published to specific topics. Consumers retrieve data from the queue and perform corresponding processing. The IoTDB subscription client provides two types of consumers: pull consumer and push consumer. - Consumer Group is a collection of consumers. When different consumers in the same consumer group subscribe to the same topic, these consumers share the processing progress of data under this topic. Each data under this topic can only be processed by one consumer within the group, ensuring that data is not processed repeatedly. Based on these concepts, the IoTDB subscription client provides a series of SDKs for creating topics, creating consumers, subscribing to topics, consuming data, committing consumption progress, and obtaining subscription relationships. Here's a comprehensive example[1] demonstrating how to use the subscription client JAVA SDK to consume data from IoTDB. Technically, the data subscription client will rely on IoTDB's existing streaming processing framework (Pipe). Each subscription corresponds to a user-invisible pipe task. Subscription relationships and other metadata information are persistently maintained through the config node. Basic functionality has been developed on the master branch[2], and further iterations will continuously improve it. I hope you are interested in this feature and would like to participate in the development and testing. You can also leave your comments and suggestions in this thread. Appreciate any suggestion/feedback & contribution. Thank you for your attention and support. Best regards, VGalaxies Reference: 1. https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/SubscriptionSessionExample.java 2. https://github.com/apache/iotdb/tree/master
