Hi, Thanks Dawid and Florin.
To Dawid: CsvTableSource doesn't implements DefinedProctimeAttribute and DefinedRowtimeAttributes interfaces, so we can not use proctime and rowtime in source ddl. Except csv, we also need to consume json and pb data. To Florin: Installing local kafka and zk introduces too many third-party components and may be not universal. In my scenario, I need to run a local sql job to debug(for example source and sink are kafka-json, dimension table is jdbc) before submit it to yarn. The following usage is what I want: 1）generate local json data for source and dimension table (source table supports proctime and rowtime); 2) replace `connetor.type` to 'filesystem'; 3) add `connector.path` to source table /dimension table ddl property; 4) new sql can run locally as data read from kafka and jdbc. Thanks, Anyang Spico Florin <[email protected]> 于2019年10月29日周二 下午6:35写道： > Hi! > > Another solution would be to locally install kafka+zookeeper and push > your dumped json (from the production server) data in a topic(you create a > Kafka producer). > Then you configure your code to point to this local broker. Consume your > data from topic from either strategy you need (earliest offset, latest). > The advantage is that you can repeat your tests multiple times as in real > scenario. > > Depending on your use case, there can be different behaviour of your > processing pipeline when you consume from a file (batch) or from a stream > (kafka). > I had this kind of issue when some CEP functionalities. > I hope it helps. > Regards, > Florin > > > On Tue, Oct 29, 2019 at 12:00 PM Anyang Hu <[email protected]> wrote: > >> Hi guys, >> >> In flink1.9, we can set `connector.type` to `kafka` and `format.type` to >> json to read/write json data from kafka or write json data to kafka. >> >> In my scenario, I wish to read local json data as a souce table, since I >> need to do local debug and don't consume online kafka data. >> >> For example: >> >>> create table source ( >>> first varchar, >>> id int >>> ) with ( >>> 'connector.type' = 'filesystem', >>> 'connector.path' = '/path/to/json', >>> 'format.type' = 'json' >>> ) >> >> >> In addition, writing local json data is also needed. >> >> Does anyone have similar needs? >> >> Best regards, >> Anyang >> >
