JunWang222 commented on code in PR #787: URL: https://github.com/apache/wayang/pull/787#discussion_r3792528814
########## guides/cost-profiling.md: ########## @@ -0,0 +1,430 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Cost Profiling Guide + +This document explains why Apache Wayang needs platform-specific cost +profiling, how profiling data is collected, how the genetic optimizer learns +cost parameters, and how users can repeat the profiling workflow on their own +hardware. + +The examples below use Trino, but the same workflow also applies to other +JDBC-based platforms such as Presto and BigQuery. + +Version 2.0 uses S01 through S16 as the profiling workload, including the +join-heavy pipelines S14 through S16. It keeps the guide focused on data +collection and parameter learning, leaving follow-up quality checks out of +scope for now. + +## 1. Why Profiling Is Needed + +Wayang can map the same logical plan to different execution platforms, such as +Java, Spark, Trino, Presto, or BigQuery. For example, a user query may contain: + +```text +TableSource -> Filter -> Projection -> TableSink +``` + +The optimizer needs a cost model to decide whether these operators should stay +on a SQL platform or be moved to another platform. In this context, "cost" does +not mean cloud billing cost. It is the numerical value that Wayang uses to +compare alternative execution plans. + +With the default Trino configuration: + +```properties +wayang.trino.costs.fix = 0.0 +wayang.trino.costs.per-ms = 1.0 +``` + +the optimizer cost can be interpreted approximately as: + +```text +cost = estimated execution time in milliseconds +``` + +However, the real execution time depends on the user's machine, cluster size, +network, database configuration, and workload. Therefore, users should profile +their own environment when they need accurate cost parameters. + +## 2. Load Profile Formulas + +Each execution operator has a load profile. For example, a table source may use +a formula like: + +```properties +wayang.trino.tablesource.load = { + "type":"mathex", + "in":0, + "out":1, + "cpu":"((10)*(out0))+(800000)", + "ram":"0", + "disk":"0", + "net":"0", + "p":0.9 +} +``` + +This can be read as: + +```text +CPU load = alpha * number_of_rows + beta +``` + Review Comment: I will add the number in the example, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
