Hello Roman, Thanks for your response.
I think partitioning you described (event type + protocol type) is subject to data skew. Including a device ID should solve this problem. Also, including "protocol_type" into the key and having topic per protocol_type seems redundant. Each protocol is in single topic and event_type is key to distribute data to a specific partition. Furthermore, do you have any particular reason to maintain multiple topics? I could imagine protocols have different speeds or other characteristics, so you can tune Flink accordingly. Otherwise, having a single topic partitioned only by device ID would simplify deployment and reduce data skew. Yes, you are right. These protocols have separate characteristics like speed, data format. If I do have only one topic with data partitioned by device_id then it could be that events from faster protocol is processed faster and the joins which I want to do will not have enough matching data. I have a question here how are you referring to tune Flink to handle different characteristics like speed of streams as reading from kafka could result in uneven processing of data? > By consume do you mean the downstream system? My downstream is TSDB and other DBs where the data will be written to. All these is time-series data. Thanks, Hemant On Tue, May 12, 2020 at 5:28 PM Khachatryan Roman < khachatryan.ro...@gmail.com> wrote: > Hello Hemant, > > Thanks for your reply. > > I think partitioning you described (event type + protocol type) is subject > to data skew. Including a device ID should solve this problem. > Also, including "protocol_type" into the key and having topic per > protocol_type seems redundant. > > Furthermore, do you have any particular reason to maintain multiple > topics? > I could imagine protocols have different speeds or other characteristics, > so you can tune Flink accordingly. > Otherwise, having a single topic partitioned only by device ID would > simplify deployment and reduce data skew. > > > By consume do you mean the downstream system? > Yes. > > Regards, > Roman > > > On Mon, May 11, 2020 at 11:30 PM hemant singh <hemant2...@gmail.com> > wrote: > >> Hello Roman, >> >> PFB my response - >> >> As I understand, each protocol has a distinct set of event types (where >> event type == metrics type); and a distinct set of devices. Is this correct? >> Yes, correct. distinct events and devices. Each device emits these event. >> >> > Based on data protocol I have 4-5 topics. Currently the data for a >> single event is being pushed to a partition of the kafka topic(producer key >> -> event_type + data_protocol). >> Here you are talking about the source (to Flink job), right? >> Yes, you are right. >> >> Can you also share how are you going to consume these data? >> By consume do you mean the downstream system? >> If yes then this data will be written to a DB, some metrics goes to >> TSDB(Influx) as well. >> >> Thanks, >> Hemant >> >> On Tue, May 12, 2020 at 2:08 AM Khachatryan Roman < >> khachatryan.ro...@gmail.com> wrote: >> >>> Hi Hemant, >>> >>> As I understand, each protocol has a distinct set of event types (where >>> event type == metrics type); and a distinct set of devices. Is this correct? >>> >>> > Based on data protocol I have 4-5 topics. Currently the data for a >>> single event is being pushed to a partition of the kafka topic(producer key >>> -> event_type + data_protocol). >>> Here you are talking about the source (to Flink job), right? >>> >>> Can you also share how are you going to consume these data? >>> >>> >>> Regards, >>> Roman >>> >>> >>> On Mon, May 11, 2020 at 8:57 PM hemant singh <hemant2...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I have different events from a device which constitutes different >>>> metrics for same device. Each of these event is produced by the device in >>>> interval of few milli seconds to a minute. >>>> >>>> Event1(Device1) -> Stream1 -> Metric 1 >>>> Event2 (Device1) -> Stream2 -> Metric 2 ... >>>> .............. >>>> ....... >>>> Event100(Device1) -> Stream100 -> Metric100 >>>> >>>> The number of events can go up to few 100s for each data protocol and >>>> we have around 4-5 data protocols. Metrics from different streams makes up >>>> a records >>>> like for example from above example for device 1 - >>>> >>>> Device1 -> Metric1, Metric 2, Metric15 forms a single record for the >>>> device. Currently in development phase I am using interval join to achieve >>>> this, that is to create a record with latest data from different >>>> streams(events). >>>> >>>> Based on data protocol I have 4-5 topics. Currently the data for a >>>> single event is being pushed to a partition of the kafka topic(producer key >>>> -> event_type + data_protocol). So essentially one topic is made up of many >>>> streams. I am filtering on the key to define the streams. >>>> >>>> My question is - Is this correct way to stream the data, I had thought >>>> of maintaining different topic for an event, however in that case number of >>>> topics could go to few thousands and that is something which becomes little >>>> challenging to maintain and not sure if kafka handles that well. >>>> >>>> I know there are traditional ways to do this like pushing it to >>>> timeseries db and then joining data for different metric but that is >>>> something which will never scale, also this processing should be as >>>> realtime as possible. >>>> >>>> Are there better ways to handle this use case or I am on correct path. >>>> >>>> Thanks, >>>> Hemant >>>> >>>
