[GitHub] [incubator-hudi] vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool
vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool URL: https://github.com/apache/incubator-hudi/pull/1362#issuecomment-601469123 No. thank you.. This kind of stuff, gives me energy to keep pushing more :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool
vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool URL: https://github.com/apache/incubator-hudi/pull/1362#issuecomment-601468133 > Do you see any other use case the reverse search would be useful? No. not at the moment.. We can close this PR out if you agree This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool
vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool URL: https://github.com/apache/incubator-hudi/pull/1362#issuecomment-601321422 Given that, do we still need the ability to search for the checkpoints in reverse time order? tbh I don't see a value in it, since there cannot be multiple writers to a hudi table anyway. May be we can think about an `CheckPointProvider` abstraction where if DeltaStreamer cannot find a checkpoint from the last delta commit/commit, it invokes `checkpointProvider.getCheckpoint()`. We can actually introduce that in this PR and have two implementations 1) (default)NoOpCheckpointProvider (throws an error if it cannot find a checkpoint) 2) ScanOlderCommitsCheckpointProvider (what you have now) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool
vinothchandar commented on issue #1362: [WIP]HUDI-644 Implement checkpoint generator helper tool URL: https://github.com/apache/incubator-hudi/pull/1362#issuecomment-599719902 @garyli1019 I understand what you are getting at.. We had a similar issue cutting over pipelines and we handled that by having ability to force a checkpoint for a single run of delta streamer.. So, I my guess is, we will explore a way to generate checkpoints from different other mechanisms like connect-hdfs.? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
