Yes correct, I set it to batch interval = 1 and it works fine. Anyways I think the JDBC sink could have some improvements like batchInterval + time interval execution. So if the batch doesn't fill up then execute what ever is left on that time interval.
On Thu, 17 Oct 2019 at 12:22, Rong Rong <walter...@gmail.com> wrote: > Hi John, > > You are right. IMO the batch interval setting is used for increasing the > JDBC execution performance purpose. > The reason why your INSERT INTO statement with a `non_existing_table` the > exception doesn't happen is because the JDBCAppendableSink does not check > table existence beforehand. That being said it should fail at the first > batch execution. > > Also I think the `batchInterval` setting is local to the task , this means > the default 5000 batchInterval is per-partition. > > -- > Rong > > On Wed, Oct 16, 2019 at 7:21 AM John Smith <java.dev....@gmail.com> wrote: > >> Ok I think I found it. it's the batch interval setting. From what I see, >> if we want "realtime" stream to the database we have to set it to 1 other >> wise the sink will wait until, the batch interval count is reached. >> >> The batch interval mechanism doesn't see correct? If the default size is >> 5000 and you need to insert 5001 you will never get that 1 record? >> >> On Tue, 15 Oct 2019 at 15:54, John Smith <java.dev....@gmail.com> wrote: >> >>> Hi, using 1.8.0 >>> >>> I have the following job: https://pastebin.com/ibZUE8Qx >>> >>> So the job does the following steps... >>> 1- Consume from Kafka and return JsonObject >>> 2- Map JsonObject to MyPojo >>> 3- Convert The stream to a table >>> 4- Insert the table to JDBC sink table >>> 5- Print the table. >>> >>> - The job seems to work with no errors and I can see the row print to >>> the console and I see nothing in my database. >>> - If I put invalid host for the database and restart the job, I get a >>> connection SQLException error. So at least we know that works. >>> - If I make a typo on the INSERT INTO statement like INSERTS INTO >>> non_existing_table, there are no exceptions thrown, the print happens, the >>> stream continues to work. >>> - If I drop the table from the database, same thing, no exceptions >>> thrown, the print happens, the stream continues to work. >>> >>> So am I missing something? >>> >>
