Re: JDBC poll operator performance

2017-06-27 Thread Thomas Weise
Records can be distributed between partitions based on key ranges, no sorting is needed for that. You may need sorting for repeatable read within a partition. But even then the query should filter to not fetch what was already loaded. Without a WHERE clause, there is an unnecessary repeated full

Re: JDBC poll operator performance

2017-06-27 Thread Hitesh Kapoor
I agree with Bhupesh, DB does not guarantees that your data will be retrieved in a specific or sorted order if an 'order by' clause is not given in the query. IMO in case of our poll operator we will have to sort the records for non-poller partitions to ensure all records are emitted and no 2

JDBC poll operator performance

2017-06-26 Thread Thomas Weise
Hi, It seems the poll operator performs unnecessary operations in the case where the "key" column values in the source table are monotonic increasing. There should be no need to sort or do count selects. Instead it should be sufficient to just filter with the key range. Let's say the key column