eladkal commented on code in PR #71976:
URL: https://github.com/apache/airflow/pull/71976#discussion_r3872155783
##########
providers/influxdb/docs/operators/index.rst:
##########
@@ -46,3 +46,24 @@ Example usage:
:language: python
:start-after: [START howto_operator_influxdb3]
:end-before: [END howto_operator_influxdb3]
+
+Deferrable mode
+^^^^^^^^^^^^^^^
+
+Set ``deferrable=True`` to release the worker slot while the query runs. The
task is resumed by the
+:class:`~airflow.providers.influxdb.triggers.influxdb3.InfluxDB3QueryTrigger`
once results are ready.
+
+.. exampleinclude:: /../../influxdb/tests/system/influxdb/example_influxdb3.py
+ :language: python
+ :start-after: [START howto_operator_influxdb3_deferrable]
+ :end-before: [END howto_operator_influxdb3_deferrable]
+
+.. note::
+
+ InfluxDB 3 streams query results over a single Arrow Flight call rather
than exposing a job that
+ can be polled, so the trigger awaits the query once instead of polling at
an interval, and there
+ is no ``poll_interval`` parameter. Results still travel back through XCom,
so deferring is most
+ useful for long-running queries with small-to-moderate result sets. For
very large extracts,
+ keep using
:class:`~airflow.providers.influxdb.hooks.influxdb3.InfluxDB3Hook` from a
Python task.
+
+ Deferrable mode requires ``influxdb3-python>=0.12.0`` and a running
``triggerer``.
Review Comment:
can be removed
##########
providers/influxdb/src/airflow/providers/influxdb/operators/influxdb3.py:
##########
@@ -40,6 +40,11 @@ class InfluxDB3Operator(BaseOperator):
:param sql: The SQL query to be executed
:param influxdb3_conn_id: Reference to :ref:`InfluxDB 3 connection id
<howto/connection:influxdb3>`.
+ :param deferrable: Run the query from the triggerer instead of holding a
worker slot for its
+ duration. Requires ``influxdb3-python>=0.12.0``. Note that InfluxDB 3
streams results over
+ Arrow Flight rather than exposing a job that can be polled, so the
whole result set still flows back
+ through XCom -- deferring helps with long-running queries returning
modest result sets
+ (aggregations, freshness probes), not with very large extracts.
Review Comment:
Please cleanup here. Explain just what is needed.
Requires ``influxdb3-python>=0.12.0`` is not helpful. The provider already
set this min version so users don't need to make sure they hae it.
To be honest I am not sure I understand what this warning means.
##########
providers/influxdb/docs/operators/index.rst:
##########
@@ -46,3 +46,24 @@ Example usage:
:language: python
:start-after: [START howto_operator_influxdb3]
:end-before: [END howto_operator_influxdb3]
+
+Deferrable mode
+^^^^^^^^^^^^^^^
+
+Set ``deferrable=True`` to release the worker slot while the query runs. The
task is resumed by the
+:class:`~airflow.providers.influxdb.triggers.influxdb3.InfluxDB3QueryTrigger`
once results are ready.
+
+.. exampleinclude:: /../../influxdb/tests/system/influxdb/example_influxdb3.py
+ :language: python
+ :start-after: [START howto_operator_influxdb3_deferrable]
+ :end-before: [END howto_operator_influxdb3_deferrable]
+
+.. note::
+
+ InfluxDB 3 streams query results over a single Arrow Flight call rather
than exposing a job that
+ can be polled, so the trigger awaits the query once instead of polling at
an interval, and there
Review Comment:
This seems to be a statement over how Influxdb operate. Can you please
update the info with a link to the Influx docs? This is needed to confirm that
are assumptions are right and in case that Influx change how it works we can
track the change/
##########
providers/influxdb/src/airflow/providers/influxdb/hooks/influxdb3.py:
##########
@@ -49,6 +50,15 @@
from airflow.models import Connection
+class InfluxDB3AsyncQueryNotAvailableError(RuntimeError):
+ """Raised when the installed InfluxDB 3 client lacks async query
support."""
+
+
+def convert_dataframe_to_records(dataframe: pd.DataFrame) -> list[dict[str,
Any]]:
Review Comment:
```suggestion
def _convert_dataframe_to_records(dataframe: pd.DataFrame) -> list[dict[str,
Any]]:
```
Lets please mark this as private. This is not a function we want users to
rely on as it's not part of the public interface we serve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
