etr2460 commented on issue #10139: URL: https://github.com/apache/incubator-superset/issues/10139#issuecomment-650258941
I started working on this today, but I don't think it makes a lot of sense to add a backoff within Superset if pyhive only supports constant polling intervals. Therefore, I have a few options, ranked in my preference order: 1. Update the sleep within handle_cursor to be equal to the poll_interval configured in the presto database's `engine_params` object. then both params will be aligned and fully configurable by the admin 2. Open a PR against pyhive to support using a generator function as well as a constant value for `poll_interval`, then update Superset to default to some variety of fibo backoff for both 3. Manually update the constant sleep value in Superset to something smaller than 1 (maybe 0.5 for now) and also default the presto engine params to also send 0.5. This isn't configurable, but it would have immediate performance improvements @john-bodley @bkyryliuk @ktmud @villebro @tooptoop4, thoughts? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
