Hi VGalaxies, The data subscription client is really a great feature. It provides users with a brand new way of consuming data, and offers new possibilities for real-time computation of industrial data. I am also pleased to see that IoTDB's stream computing engine has new usage scenarios.
Thank you for your contribution. I hope you can continue to improve the data subscription feature and make it more user-friendly. Thanks, Steve On 2024/04/08 03:06:59 VGalaxies wrote: > Hello everyone, > > I am VGalaxies, a new contributor to Apache IoTDB. I am excited to > share with you a new feature that I have been working on for the past > few months. > > The data subscription client is a new way to access data within IoTDB, > distinct from the traditional method of querying data using SQL-like > syntax. In scenarios where real-time data, quick response to data > changes, and building highly event-driven systems are required, data > subscription has greater advantages over data querying. For example, > in the following two scenarios: > > 1. Replace extensive polling queries for large amounts of data: Avoid > significant impacts on the performance of existing systems when > querying frequently or when there are many data points. Also, avoid > problems with determining the query scope and ensure downstream > receives accurate full data. > 2. Facilitate downstream system integration: It's easier to integrate > with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc. > There's no need to customize the logic of IoTDB's data change capture > for each big data component separately, simplifying integration > component design and making it easier for users. > > The IoTDB subscription client references some features defined by some > message queue products like Kafka. It consists of 3 core concepts: > Topic, Consumer, and Consumer Group. > > - Topic is a logical concept used by the IoTDB subscription client to > classify data, serving as a channel for data publication. Producers > publish data to specific topics, while consumers subscribe to these > topics to receive related data. In the IoTDB subscription client, > topics describe the sequence characteristics, time characteristics, > presentation forms, and optional custom processing logic of subscribed > data. > - Consumer is an application or service in the IoTDB subscription > client responsible for receiving and processing data published to > specific topics. Consumers retrieve data from the queue and perform > corresponding processing. The IoTDB subscription client provides two > types of consumers: pull consumer and push consumer. > - Consumer Group is a collection of consumers. When different > consumers in the same consumer group subscribe to the same topic, > these consumers share the processing progress of data under this > topic. Each data under this topic can only be processed by one > consumer within the group, ensuring that data is not processed > repeatedly. > > Based on these concepts, the IoTDB subscription client provides a > series of SDKs for creating topics, creating consumers, subscribing to > topics, consuming data, committing consumption progress, and obtaining > subscription relationships. Here's a comprehensive example[1] > demonstrating how to use the subscription client JAVA SDK to consume > data from IoTDB. > > Technically, the data subscription client will rely on IoTDB's > existing streaming processing framework (Pipe). Each subscription > corresponds to a user-invisible pipe task. Subscription relationships > and other metadata information are persistently maintained through the > config node. Basic functionality has been developed on the master > branch[2], and further iterations will continuously improve it. > > I hope you are interested in this feature and would like to > participate in the development and testing. You can also leave your > comments and suggestions in this thread. Appreciate any > suggestion/feedback & contribution. > > Thank you for your attention and support. > > Best regards, > > VGalaxies > > Reference: > 1. > https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/SubscriptionSessionExample.java > 2. https://github.com/apache/iotdb/tree/master >
