betodealmeida commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r808364795
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`,
"
+ "which are required to be installed in your environment in
order "
+ "to estimate cost"
Review comment:
Curious, wouldn't these be necessarily installed if the user has a
BigQuery database connected?
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`,
"
+ "which are required to be installed in your environment in
order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials =
service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else
str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
I think we can change the `humanize` function in Presto (it's also
duplicated in Trino) to return bytes in 1024 increments. The only reason I did
1000 is because it's also applied to row count and other parameters. This way
it's consistent. Ideally we'd have a single function used by Presto, Trino,
BigQuery and other engine specs.
(Note that it's also possible to overwrite the formatter function using
`QUERY_COST_FORMATTERS_BY_ENGINE` in the config. We used that at Lyft to show
the query cost in dollars, estimated run time, and carbon footprint.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
