vinothchandar commented on issue #917: [HUDI-251] JDBC incremental load to HUDI with DeltaStreamer URL: https://github.com/apache/incubator-hudi/pull/917#issuecomment-534684983 >So here is the kind of algorithm that I think of implementing steps 1 & 2 sound good to me. >>but user mentions 5 mins interval that means jobs will keep on piling right? next batch wont be schedule until the first one completes. so there is backpressure there already to prevent pile up. >>spark.read().jdbc("url,"table","someTableColumn",1,10,connectionProps) does not spark not already parallelize the pull? i.e allocate 1-2 to one executor, 2-3 to another and so on?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
