oleksiidav commented on code in PR #33472:
URL: https://github.com/apache/airflow/pull/33472#discussion_r1307310732
##########
airflow/providers/databricks/hooks/databricks.py:
##########
@@ -165,17 +165,28 @@ def list_jobs(
:param offset: The offset of the first job to return, relative to the
most recently created job.
:param expand_tasks: Whether to include task and cluster details in
the response.
:param job_name: Optional name of a job to search.
+ :param page_token: The optional page token pointing at the first first
job to return.
:return: A list of jobs.
"""
has_more = True
all_jobs = []
-
+ use_token_pagination = (page_token is not None) or (offset is None)
+ if offset is not None:
+ print("[WARN] You are using the offset parameter in list_jobs
which will be deprecated soon")
Review Comment:
It follows from [this
discussion](https://github.com/apache/airflow/pull/33472#discussion_r1297282608)
- the parameter has been deprecated by Databricks already, and the plan would
be to show a deprecation warning in Airflow once this PR is merged, and then
remove it from the hook altogether after 2 releases (around 1 month).
If we don't do this by Oct, and the Airflow hook still uses the offset
param, then Databricks-enforced API limits will kick in, resulting in many
offset-based requests possibly failing, which would be a worse way of handling
a breaking change than deprecating it softly with an intermediate warning.
Let's drive the switch to token-based pagination asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
