vogievetsky commented on code in PR #12983: URL: https://github.com/apache/druid/pull/12983#discussion_r957900529
########## docs/multi-stage-query/msq-durable-storage.md: ########## @@ -0,0 +1,63 @@ +--- +id: durable-storage +title: Enable durage storage for mesh shuffle for the MSQ task engine +sidebar_label: Durable storage +description: Using durable storage for mesh shuffle can improve the reliability of your SQL-based ingestion at the cost of some performance. +--- + +> SQL-based ingestion using the multi-stage query task engine is our recommended solution starting in Druid 24.0. Alternative ingestion solutions, such as native batch and Hadoop-based ingestion systems, will still be supported. We recommend you read all [known issues](./msq-known-issues.md) and test the feature in a development environment before rolling it out in production. Using the multi-stage query task engine with `SELECT` statements that do not write to a datasource is experimental. + +By default, the multi-stage query task engine (MSQ task engine) uses the local storage of a node to store data from intermediate steps when executing a query. Although this method provides better speed when executing a query, the data is lost if the node encounters an issue. When you enable durable storage, intermediate data is stored in Amazon S3 instead. Using this feature can improve the reliability of queries that use more than 20 workers. In essence, you trade some performance for better reliability. This is especially useful for long-running queries. Review Comment: Where does the "20 workers" language come from? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
