mcvsubbu commented on issue #6302: URL: https://github.com/apache/pinot/issues/6302#issuecomment-892265494
> There is an API (check Task section on swagger) to trigger the movement from real-time to offline instead of waiting for the periodic task (you still need minions). However the realtimeToOffline task is not designed to achieve exactly what you want. It will not pick up the segments for movement unless they're completed segments. If it seems consuming segments, it will wait for them to complete. https://youtu.be/V_KNUUrS6DA (minute 33 onwards) explains that behavior. We might need to change the job's behavior to support this. > […](#) > On Tue, Aug 3, 2021, 7:04 AM Valentin Touffet ***@***.***> wrote: I would prefer using API instead of minions, since I would do it as soon as my job finished processing data, I want to end this job only when the data is ready (which means when the segment is in the OFFLINE table in this solution). Not an issue if the API supports it since we can automate it in our job. This solution seems to work for our use case. But downloading and uploading segments could be a long task, isn't it? There is no other way? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <[#6302 (comment)](https://github.com/apache/pinot/issues/6302#issuecomment-892012040)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWIC3NWSBA6K6TAQS5ZSEDT3AOQLANCNFSM4UIMSSVQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.g oogle.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> . The segment will complete when the time constraint is reached. Since there is no more new data, the last completed segment will be moved, and the next consuming segment will be empty. Am I right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
