gopidesupavan commented on code in PR #62232: URL: https://github.com/apache/airflow/pull/62232#discussion_r2839731815
########## providers/common/ai/src/airflow/providers/common/ai/datafusion/engine.py: ########## @@ -0,0 +1,102 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from functools import cached_property +from typing import TYPE_CHECKING, Any + +from datafusion import SessionContext + +from airflow.providers.common.ai.datafusion.format_handlers import get_format_handler +from airflow.providers.common.ai.datafusion.object_storage_provider import ObjectStorageProviderFactory +from airflow.providers.common.ai.exceptions import ObjectStoreCreationException, QueryExecutionException +from airflow.providers.common.ai.utils.config import ConnectionConfig, DataSourceConfig +from airflow.utils.log.logging_mixin import LoggingMixin + + +class DataFusionEngine(LoggingMixin): + """Apache DataFusion engine.""" + + def __init__(self): + super().__init__() + # TODO: session context has additional parameters via SessionConfig see what's possible we can use Possible via DataFusionHook ? + self.df_ctx = SessionContext() + self.registered_tables: dict[str, str] = {} + + @cached_property + def session_context(self) -> SessionContext: + """Return the session context.""" + return self.df_ctx Review Comment: updated :) i overthink ########## providers/common/ai/docs/operators/analytics.rst: ########## @@ -0,0 +1,82 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Analytics Operator +=================== + +The Analytics operator is designed to run analytic queries on data stored in various datastores. It is a generic operator that can query data in S3, GCS, Azure, and Local File System. + +The Analytics Operator uses Apache DataFusion as its query engine and supports SQL as the query language. It operates on a single node engine to deliver high-performance analytics on the data. It can be used for various analytics tasks such as data exploration, data aggregation, and more. `<https://datafusion.apache.org/>`_. + + +When to Use Analytics Operator +------------------------------ + +Analytics Operator is suitable for running analytics on large volumes of datasets, with performance and efficiency. Under the hood, it uses Apache DataFusion, a high-performance, extensible query engine for Apache Arrow, which enables fast SQL queries on various data formats and storage systems. DataFusion is chosen for its ability to handle large-scale data processing on a single node, providing low-latency analytics without the need for a full database setup and without the need for high compute clusters. For more on Analytics Operator with DataFusion use cases, see `<https://datafusion.apache.org/user-guide/introduction.html#use-cases>`_. + + +Supported Storage Systems +------------------------- +- S3 +- Local File System Review Comment: updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
