Hi, some update on this topic,
After studying the protobuf , I found out I can't compress a JSON
(beginner's mistake)
The following will generate a valid remote write HTTP post:
1. Download the proto files from
https://github.com/prometheus/prometheus/tree/master/prompb (also need
goto.proto, and google.protobuf packages, so this can be handy:
https://github.com/rgordill/prometheus-kafka-remote-writer-sb/tree/master/src/main/proto
)
2. Since I am using java, there is some gradle plugin can help compiling
proto file into java: https://github.com/google/protobuf-gradle-plugin
3. Construct a "RemoteWrite" object using those generated java classes
(Builder) and use snappy to compress the ByteArray from the RemoteWrite
object... post it through the HttpRequest .... Bingo, the remote write went
over the wire and it shows up in InfluxDB (it listens at
http://x.x.x.x:8086/api/v1/prom/write?db=prometheus ) A small trick is to
set metrics name "up" is to create a Label like ("__name__","up") in the
timeseries.
Cheers,
Qiang
On Sunday, April 19, 2020 at 2:58:15 PM UTC-5, Qiang Ma wrote:
>
> Thank you for the response!
>
> My code does have the headers set , see below:
>
>
> byte[] compressed = Snappy.compress(payload.getBytes("UTF-8"));
> LOGGER.log(Level.INFO, "Sending ="+ url + " with: \n"+ new
> String(Snappy.uncompress(compressed)));
> HttpClient client = HttpClient.newHttpClient();
> HttpRequest request = HttpRequest.newBuilder()
> .uri(URI.create(url))
> .setHeader("Content-Type", "application/x-protobuf")
> .header("Content-Encoding", "snappy")
> .setHeader("X-Prometheus-Remote-Write-Version", "0.1.0")
> .header("User-Agent", "userAgent")
> .POST(HttpRequest.BodyPublishers.ofByteArray(compressed))
> .build();
>
>
> Also, thanks for pointing out the wire type 6 is not even listed... maybe
> I will set up a different remote_write target to try out. Or I can contact
> influx DB to see whether they extended wire type 6 for something else :)
>
> Regards,
> Qiang
>
> On Sunday, April 19, 2020 at 2:31:07 PM UTC-5, Julius Volz wrote:
>
>> Are you setting the right headers (esp. "Content-Encoding: snappy") so
>> that the remote end knows the request is snappy-compressed? Like these here:
>>
>>
>> https://github.com/prometheus/prometheus/blob/cd12f0873c3eb2031f7ba9b2e169449aa1012e3f/storage/remote/client.go#L84-L87
>>
>> Because your error is a protobuf decoding error referring to a wire type
>> which protobuf itself does not define (max wire type is 5, see
>> https://developers.google.com/protocol-buffers/docs/encoding#structure),
>> maybe the receiving end is trying to interpret the request as protobuf
>> without decompressing it first?
>>
>> On Sun, Apr 19, 2020 at 3:52 PM Qiang Ma <[email protected]> wrote:
>>
>>> Hi, All,
>>>
>>> I am trying to write a Java client to push metrics through remote_write
>>> (as I want to use PromQL to narrow down number of metrics).
>>>
>>> The receiving end is an InfluxDB and it is taking data from prometheus
>>> native "remote_write" with no problem.
>>>
>>> But when I use the Java client to post the payload (encoded with
>>> org.xerial.snappy.Snappy) and the client got this error:
>>> {"error":"proto: illegal wireType 6"}
>>>
>>> There are not much I could find how to trouble shoot to understand
>>> what's the problem... hoping there is someone in this group can lend me
>>> some tips!
>>> Thanks in advance!
>>>
>>> Regards,
>>> Qiang
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/b3c9aaab-d85b-4b81-a922-2c2032916aac%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/prometheus-users/b3c9aaab-d85b-4b81-a922-2c2032916aac%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8a9f5371-2f14-4475-98da-56efcdd259a0%40googlegroups.com.
