Re: Converting streaming to batch execution

2019-11-27 Thread Caizhi Weng
Hi Nick, It seems to me that the slow part of the whole pipeline is the Derby sink. Could you change it into other sinks (for example, csv sink or even a "discard everything" sink) and see if the throughput improves? If this is the case, are you using the JDBC connector? If yes, you might conside

Converting streaming to batch execution

2019-11-27 Thread Nicholas Walton
Hi, I’ve been working with a pipleline that was initially aimed at processing high speed sensor data, but for a proof of concept I’m feeding simulated data from a CSV file. Each row of the file is a sample across a number of time series, and I’ve been using the streaming environment to process