Re: Integrate IoTDB as a data source in StreamPipes

[Philipp Zehnder]

Hi Steve,

thanks for your answer.

Actually I think it is a good idea to use a message broker, this way a lot of 
functionality can be re-used.

Do you already have a concept, how to register such a broker?
Will this be done directly in IoTDB or will there be an API that a client can 
use to register a message broker for a certain time-series?
If a client could register a broker to IoTDB,  we could use the internal MQTT 
broker of StreamPipes or even implement our own solution with (e.g. [1]).

A StreamPipes adapter would then register a broker and IoTDB could forward the 
data to this broker from whom we subscribe to the events.
Alternatively, we could directly subscribe to the Broker and the user would 
have to configure IoTDB manually.

What do you think?

Philipp

[1] https://github.com/moquette-io/moquette 


> On 18. May 2020, at 18:04, Steve Yurong Su  wrote:
> 
> Hi Philipp,
> 
> Many thanks for your reply.
> 
> Considering system performance and other aspects, IoTDB is not prepared to
> support external systems to directly subscribe to data changes in the
> database. As a result, we may not have a Java client for StreamPipes to
> consume events.
> 
> On the other hand, we hope that external systems can simply accept our
> event push. Therefore, a better option for us is that we push events to the
> brokers, and then StreamPipes consumes them from the brokers.
> 
> If you have a better idea, please let me know.
> 
> Best,
> Steve
> 
> Philipp Zehnder  于 2020年5月11日周一 下午1:03写道：
> 
>> Hi Steve,
>> 
>> thanks for your mail.
>> Yes, most of the adapters are pull bases, but StreamPipes also supports
>> data sources that push events.
>> For an example you can have a look at the broker adapters (e.g. MQTT [1],
>> Kafka [2], Pulsar [3]).
>> 
>> We could implement a similar solution for IoTDB, where we use a client
>> library to consume the events.
>> How can we consume events produced in IoTDB in Java? Did you already
>> implement such a client library?
>> 
>> Please ask if you have any further questions or if you need help.
>> I could for example provide you a template class with all the StreamPipes
>> related API code. Then you can focus on the IoTDB integration part.
>> 
>> Philipp
>> 
>> 
>> [1]
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/MqttProtocol.java
>> <
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/MqttProtocol.java
>>> 
>> [2]
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/KafkaProtocol.java
>> <
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/KafkaProtocol.java
>>> 
>> [3]
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/pulsar
>> <
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/pulsar
>>> 
>> 
>> On 2020/05/10 16:36:39, Steve Yurong Su  wrote:
>>> Hi,>
>>> 
>>> I'm Steve and I come from the IoTDB community. As we discussed with
>> Philipp>
>>> at [1], we really wanted to make IoTDB a data source in StreamPipes.>
>>> 
>>> I have read the codes of the adapters in StreamPipes, and it seems that
>> all>
>>> adapters are getting data from external systems by "pulling". Then my>
>>> question is, if we don't want to rely on broker technologies, or any
>> APIs>
>>> in our system, is there any other way to transfer data to StreamPipes>
>>> directly? For example, does StreamPipes have an API that allows
>> external>
>>> systems to register a source and allows external systems to push data
>> to>
>>> StreamPipes?>
>>> 
>>> Since I am not familiar with the source codes of StreamPipes, I hope to
>> get>
>>> your help :)>
>>> 
>>> [1]>
>>> 
>> https://lists.apache.org/thread.html/rcbe96b4f4ee1ec0b6f774ec03e962090aca290bd153d6ff9cf3ea96a%40%3Ciot.apache.org%3E>
>> 
>>> 
>>> Best,>
>>> Steve>
>>> 
>> 
>> 
>> 

Re: Integrate IoTDB as a data source in StreamPipes

[Steve Yurong Su]

Hi Philipp,

Many thanks for your reply.

Considering system performance and other aspects, IoTDB is not prepared to
support external systems to directly subscribe to data changes in the
database. As a result, we may not have a Java client for StreamPipes to
consume events.

On the other hand, we hope that external systems can simply accept our
event push. Therefore, a better option for us is that we push events to the
brokers, and then StreamPipes consumes them from the brokers.

If you have a better idea, please let me know.

Best,
Steve

Philipp Zehnder  于 2020年5月11日周一 下午1:03写道：

> Hi Steve,
>
> thanks for your mail.
> Yes, most of the adapters are pull bases, but StreamPipes also supports
> data sources that push events.
> For an example you can have a look at the broker adapters (e.g. MQTT [1],
> Kafka [2], Pulsar [3]).
>
> We could implement a similar solution for IoTDB, where we use a client
> library to consume the events.
> How can we consume events produced in IoTDB in Java? Did you already
> implement such a client library?
>
> Please ask if you have any further questions or if you need help.
> I could for example provide you a template class with all the StreamPipes
> related API code. Then you can focus on the IoTDB integration part.
>
> Philipp
>
>
> [1]
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/MqttProtocol.java
> <
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/MqttProtocol.java
> >
> [2]
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/KafkaProtocol.java
> <
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/KafkaProtocol.java
> >
> [3]
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/pulsar
> <
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/pulsar
> >
>
> On 2020/05/10 16:36:39, Steve Yurong Su  wrote:
> > Hi,>
> >
> > I'm Steve and I come from the IoTDB community. As we discussed with
> Philipp>
> > at [1], we really wanted to make IoTDB a data source in StreamPipes.>
> >
> > I have read the codes of the adapters in StreamPipes, and it seems that
> all>
> > adapters are getting data from external systems by "pulling". Then my>
> > question is, if we don't want to rely on broker technologies, or any
> APIs>
> > in our system, is there any other way to transfer data to StreamPipes>
> > directly? For example, does StreamPipes have an API that allows
> external>
> > systems to register a source and allows external systems to push data
> to>
> > StreamPipes?>
> >
> > Since I am not familiar with the source codes of StreamPipes, I hope to
> get>
> > your help :)>
> >
> > [1]>
> >
> https://lists.apache.org/thread.html/rcbe96b4f4ee1ec0b6f774ec03e962090aca290bd153d6ff9cf3ea96a%40%3Ciot.apache.org%3E>
>
> >
> > Best,>
> > Steve>
> >
>
>
>

Re: Integrate IoTDB as a data source in StreamPipes

[Philipp Zehnder]

Hi Steve,

thanks for your mail.
Yes, most of the adapters are pull bases, but StreamPipes also supports data 
sources that push events.
For an example you can have a look at the broker adapters (e.g. MQTT [1], Kafka 
[2], Pulsar [3]).

We could implement a similar solution for IoTDB, where we use a client library 
to consume the events.
How can we consume events produced in IoTDB in Java? Did you already implement 
such a client library? 

Please ask if you have any further questions or if you need help. 
I could for example provide you a template class with all the StreamPipes 
related API code. Then you can focus on the IoTDB integration part.

Philipp


[1] 
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/MqttProtocol.java
 

[2] 
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/KafkaProtocol.java
 

[3] 
https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/protocol/stream/pulsar
 


On 2020/05/10 16:36:39, Steve Yurong Su  wrote: 
> Hi,> 
> 
> I'm Steve and I come from the IoTDB community. As we discussed with Philipp> 
> at [1], we really wanted to make IoTDB a data source in StreamPipes.> 
> 
> I have read the codes of the adapters in StreamPipes, and it seems that all> 
> adapters are getting data from external systems by "pulling". Then my> 
> question is, if we don't want to rely on broker technologies, or any APIs> 
> in our system, is there any other way to transfer data to StreamPipes> 
> directly? For example, does StreamPipes have an API that allows external> 
> systems to register a source and allows external systems to push data to> 
> StreamPipes?> 
> 
> Since I am not familiar with the source codes of StreamPipes, I hope to get> 
> your help :)> 
> 
> [1]> 
> https://lists.apache.org/thread.html/rcbe96b4f4ee1ec0b6f774ec03e962090aca290bd153d6ff9cf3ea96a%40%3Ciot.apache.org%3E>
>  
> 
> Best,> 
> Steve> 
>