[
https://issues.apache.org/jira/browse/AIRFLOW-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938090#comment-16938090
]
ASF GitHub Bot commented on AIRFLOW-5554:
-----------------------------------------
suburbanmtman commented on pull request #6185: [AIRFLOW-5554] Require statsd
3.3.0 for timedelta stat transformation
URL: https://github.com/apache/airflow/pull/6185
Make sure you have checked _all_ steps below.
### Jira
- [ X ] My PR addresses the following [Airflow Jira]
- https://issues.apache.org/jira/browse/AIRFLOW-5554
### Description
- [ X ] Here are some details about my PR, including screenshots of any UI
changes:
When statsd is enabled in airflow, there are some calls to the statsd client
library that pass timedeltas as the duration for measuring the timing,
including dag success and failure durations.
However, handling timedeltas and converting them to millisecond floats is
not introduced until statsd 3.3.0. This should be the minimum version required
in airflow to avoid a TypeError in these stats.
### Tests
- [ X ] My PR does not need testing for this extremely good reason:
No code changes introduced, the only behavioral change is inside the statsd
library implementation due to the version change.
### Commits
- [ X ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ X ] In case of new functionality, my PR adds documentation that
describes how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Airflow 1.10.4+ needs statsd>=3.3.0
> -----------------------------------
>
> Key: AIRFLOW-5554
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5554
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun, dependencies
> Affects Versions: 1.10.4, 1.10.5
> Reporter: David Smith
> Assignee: David Smith
> Priority: Major
>
> Our environment configuration uses statsd 3.2.2, which is a valid dependency
> version choice in airflow's requirements. However if statsd is enabled in
> airflow config, there is a critical bug preventing dag runs from completing
> properly when a timedelta is passed to the statsd timing to be formatted into
> a float:
> [https://github.com/apache/airflow/blob/66a139d734bd434caa792007c3b980ca4cf8f931/airflow/models/dagrun.py#L354]
> Causing an exception to be raised due to the unexpected delta data type:
> File
> "/opt/evidation/data-processing-pipeline/lib/python3.5/site-packages/airflow/jobs/scheduler_job.py",
> line 1547, in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File
> "/opt/evidation/data-processing-pipeline/lib/python3.5/site-packages/airflow/jobs/scheduler_job.py",
> line 1242, in _process_dags
> schedule_delay)
> File
> "/opt/evidation/data-processing-pipeline/lib/python3.5/site-packages/statsd/client.py",
> line 93, in timing
> self._send_stat(stat, '%0.6f|ms' % delta, rate)
> TypeError: a float is required
> The code to detect the timedelta type and convert to millis was introduced in
> 3.3.0:
> [https://github.com/jsocol/pystatsd/blob/1c90b9fdf322680e2625da659abc2aa5d79b5bff/statsd/client/base.py#L28]
> So the setup.py in airflow should be updated to make this the minimum
> dependency:
> [https://github.com/apache/airflow/blob/66a139d734bd434caa792007c3b980ca4cf8f931/setup.py#L262]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
