GitHub user nsivabalan added a comment to the discussion: Spark query and write serving endpoint for hudi-lakehouse
thanks for your response. that mostly covers it. But I will go over the 10 bullets I have raised above to ensure we are on same page. 1. Scope of the Spark endpoint (v1 and long term) agreed. lets stick to interactive SQL. i.e. just SQL writes and reads. 2. Compute and resource management (v1 and long term) Very simple single executor pool. when the driver comes up, we define minimum and maximum for auto scaling/dynamic allocation. and thats about it. 3. Workload isolation and resiliency (v1 fundamentals) We can keep V1 simple. no special handling towards this end. 4. Operation lifecycle and retry semantics (long term) Again, for V1. we are not going to do anything. but lets keep a pin on it. on whats our plan for long term. 5. Security and multi-tenancy (long term) for V1. we are not going to do anything. but lets keep a pin on it. on whats our plan for long term. 6. Resource governance and observability (v1 fundamentals) v1 is going to be simple. For concurrent writers, lets expect users to set the right set of writer properties. 7. Extensibility and compatibility (long term) No AIs in V1. 8. Streaming and long-running jobs (long term) likely no even for long term. 9. Catalog and metadata consistency (v1 validation) Yes, we need to play this out. Atleast we should know what are the limitations if there is any. For users, do we expect them to trigger explicit refresh. 10. Hudi writer configuration (minor clarification) This will be critical to get it working. GitHub link: https://github.com/apache/hudi/discussions/19843#discussioncomment-19152297 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
