asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913587012
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -181,22 +181,26 @@ private static ByteBuf generate0(PulsarService pulsar,
boolean includeTopicMetri
//when write out 200MB data, MAX_COMPONENTS = 64 needn't mem_copy.
see: CompositeByteBuf#consolidateIfNeeded()
ByteBuf buf =
UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
boolean exceptionHappens = false;
+ //Used in namespace/topic and transaction aggregators as share metric
names
+ PrometheusMetricStreams metricStream = new PrometheusMetricStreams();
Review Comment:
`metricStream` --> `metricStreams`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -215,6 +219,7 @@ private static ByteBuf generate0(PulsarService pulsar,
boolean includeTopicMetri
//if exception happens, release buffer
if (exceptionHappens) {
buf.release();
+ metricStream.releaseAll();
Review Comment:
The `if (exceptionHappens)` is there because the function returns that
buffer hence you can't release a buffer that is about to be used. Only the
caller knows when to release it. I would fix the comment to mention that.
In your case, the creator of the `ByteBuf` inside the metric streams is you,
and it is not exposed outside, so it makes sense to release it regardless.
The original code is a bit risky since it can easily lead to a memory leak.
It's best to allocate and free at the same method, but it's beyond the scope of
this PR for sure.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together
under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new
HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE
header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String...
labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i
+ 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write('
').write(System.currentTimeMillis()).write('\n');
Review Comment:
In the original code, time was not written.
Exposition format mandates it optional, since it missing, the timestamp is
assumed to be the scraping time.
The advantage of not writing it is saving time to download the response body
to Prometheus, especially for a very large amount of topics.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together
under the same TYPE header when written.
Review Comment:
I would also add: "Since those are the requirement of the Prometheus
Exposition protocol"
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -181,22 +181,26 @@ private static ByteBuf generate0(PulsarService pulsar,
boolean includeTopicMetri
//when write out 200MB data, MAX_COMPONENTS = 64 needn't mem_copy.
see: CompositeByteBuf#consolidateIfNeeded()
ByteBuf buf =
UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
boolean exceptionHappens = false;
+ //Used in namespace/topic and transaction aggregators as share metric
names
Review Comment:
Do they share metrics? how? Each has its own prefix, no?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together
under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new
HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE
header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String...
labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i
+ 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write('
').write(System.currentTimeMillis()).write('\n');
+ }
+
+ /**
+ * Flush all the stored metrics to the supplied stream.
+ * @param stream the stream to write to.
+ */
+ void flushAllToStream(SimpleTextOutputStream stream) {
+ metricStreamMap.values().forEach(s -> stream.write(s.getBuffer()));
+ releaseAll();
Review Comment:
I would call release once in finally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
