JulianJaffePinterest commented on PR #12159:
URL: https://github.com/apache/druid/pull/12159#issuecomment-1168123015
@paul-rogers a public external task API could certainly be something worth
integrating with! I foresee a couple of solvable problems but I'd be happy to
integrate if the new API meets the connector's needs.
- I think there would need to be scalability improvements to the overlord -
suddenly dumping thousands of tasks on to the overlord while it's also running
real time ingestion has caused me trouble in the past. I'd also be a little
worried about the history footprint for these external tasks.
- I'd also be a little concerned about the change in model for failure
recovery - right now, if a Spark job hard crashes and doesn't clean up after
itself, at worst it will have left some orphaned files in deep storage that can
be easily cleared. If the job has obtained locks with Druid now a lot of
business logic needs to go in to figure out how to recover (should Spark as a
whole be re-entrant? That solves the daily batch workflow where a retry is
usually the first recovery approach but fails for something like an hourly
workflow where there may be time overruns. What about multiple jobs etc.?)
These are obviously solvable problems but they will introduce a fair amount of
orchestration complexity without an obvious place to abstract it away. If we
want to heartbeat from Spark we'll need to support a variety of network
topologies and permissions structures or else the connectors won't be useful
for a lot of people (heartbeating from Spark to an external service is a tricky
propo
sition with a lot of gotchas in my experience, and that's for single-site
uses).
Finally, as a broader note the effort to add Spark connectors to Druid is
now over two years old. I wouldn't worry too much on your side if the concerns
I've highlighted are too onerous to solve on your side or aren't where you
imagine the proposed API heading - it's unlikely these connectors will ever be
part of the main Druid code base and so I'm not sure how much effort you should
expend building against their requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
