[
https://issues.apache.org/jira/browse/FLINK-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094390#comment-17094390
]
Chesnay Schepler commented on FLINK-16611:
------------------------------------------
hmm, it is a bit of a pain to keep track of the actual size isn't it, with the
whole JSON overhead.
Let's do some math:
The limit imposed by DataDog is [3.2
megabytes|https://docs.datadoghq.com/api/?lang=bash#post-timeseries-points].
I did a small test using a metric registered for an operator (which have the
most tags), and arrived at roughly 500 bytes for the metric alone.
{code}
{"series":[{"metric":"UnregisteredHost.taskmanager.0.UnregisteredJob.UnregisteredOperator.0.testCounter","type":"gauge","host":"UnregisteredHost","tags":["","job_id:00000000000000000000000000000000","task_id:00000000000000000000000000000000","task_attempt_id:00000000000000000000000000000000","operator_id:00000000000000000000000000000000","operator_name:UnregisteredOperator","task_name:UnregisteredTask","task_attempt_num:0","job_name:UnregisteredJob","tm_id:0","subtask_index:0"],"points":[[1588068856,1]]}]}
{code}
To account for additional (possibly configured) tags and longer task/job names
etc, let's say 1 kilobyte per metric.
This gives us a theoretical limit of ~3200 metrics per request; so it should be
relatively safe to default to 2000-2500, and make it configurable as you
suggested.
> Datadog reporter should chunk large reports
> -------------------------------------------
>
> Key: FLINK-16611
> URL: https://issues.apache.org/jira/browse/FLINK-16611
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
> Affects Versions: 1.9.0
> Reporter: Chesnay Schepler
> Assignee: Stephen Whelan
> Priority: Major
>
> Datadog has a maximum size for reports that it accepts.
> If the report exceeds this size it is simply rejected, rendering the reporter
> unusable.
> We should investigate what this size limit is, and split the report into
> multiple chunks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
