Yes, I agree, thank you very much for your input. The incoming data is being treated as a string by the node AMQ stomp client library, and, attempting to convert that string to a binary array is what is not working, (thus the funky 65533 value). I either need to find a way for the AMQ client to accept in-coming data as binary or over-ride the charCodeAt to truly return a data byte. Eesh. Thanks again!
On Monday, January 11, 2021 at 2:13:59 AM UTC-7 [email protected] wrote: > hex "82 01" is indeed the expected field header for field 16, > length-prefixed (i.e. wire type 2); further, the wider hex "12 02 68 69 1a > 05 74 68 65 72 65 82 01 0c 0a 0a 6d 6f 6e 6b 65 79 62 75 74 74" is > perfectly well-formed: > > [image: image.png] > > (via https://protogen.marcgravell.com/decode) > > So: if it isn't working, there are two likely scenarios: > > 1. the serializer has a bug > 2. the data is being corrupted during processing, perhaps (usually) by > treating it as text data rather than binary (meaning: the buffer you're > passing to the serializer doesn't actually contain hex "12 02 68 69 1a 05 > 74 68 65 72 65 82 01 0c 0a 0a 6d 6f 6e 6b 65 79 62 75 74 74" > > In my experience, option 2 is *vastly* more common than option 1 > > Marc > > On Fri, 8 Jan 2021 at 15:06, peo stri tise safe <[email protected]> > wrote: > >> More info-- when the "header" is set with an index of >= 16, the encoding >> process appears to insert an extra byte in the encoded value, and a decode >> attempt will fail, here is an example where an index of 15 can allow for an >> encode and subsequent decode, but an index of 16 fails at the subsequent >> decode: >> >> using a header with an index = 15 >> obj to publish (typeof [object]): [TestResponse { >> header: { timestamp: 'monkeybutt' }, >> mykey: 'hi', >> myvalue: 'there' >> }] >> encodded as buff (typeof [object]): [<Buffer 12 02 68 69 1a 05 74 68 65 >> 72 65 7a 0c 0a 0a 6d 6f 6e 6b 65 79 62 75 74 74>] >> encoded as str (typeof [string]): >> ['\u0012\u0002hi\u001a\u0005therez\f\n\nmonkeybutt'] >> sanBinArray (typeof [object]): [[ >> 18, 2, 104, >> 105, 26, 5, >> 116, 104, 101, >> 114, 101, 122, >> 12, 10, 10, >> 109, 111, 110, >> 107, 101, 121, >> 98, 117, 116, >> 116, [length]: 25 >> ]] >> Sanity check, decoding encoded msg: (typeof [object]): [TestResponse { >> mykey: 'hi', >> myvalue: 'there', >> header: Monkey { timestamp: 'monkeybutt' } >> }] >> pinging >> CONSUMER << RXD [25] elements (typeof [string]): >> ['\u0012\u0002hi\u001a\u0005therez\f\n\nmonkeybutt'] >> As binArray (typeof [object]): [[ >> 18, 2, 104, >> 105, 26, 5, >> 116, 104, 101, >> 114, 101, 122, >> 12, 10, 10, >> 109, 111, 110, >> 107, 101, 121, >> 98, 117, 116, >> 116, [length]: 25 >> ]] >> (typeof [object]): [TestResponse { >> mykey: 'hi', >> myvalue: 'there', >> header: Monkey { timestamp: 'monkeybutt' } >> }] >> >> >> using a header with index = 16 >> obj to publish (typeof [object]): [TestResponse { >> header: { timestamp: 'monkeybutt' }, >> mykey: 'hi', >> myvalue: 'there' >> }] >> encodded as buff (typeof [object]): [<Buffer 12 02 68 69 1a 05 74 68 65 >> 72 65 82 01 0c 0a 0a 6d 6f 6e 6b 65 79 62 75 74 74>] >> >> == == <-- two bytes >> >> encoded as str (typeof [string]): >> ['\u0012\u0002hi\u001a\u0005there�\u0001\f\n\nmonkeybutt'] >> sanBinArray (typeof [object]): [[ >> 18, 2, 104, >> 105, 26, 5, >> 116, 104, 101, >> 114, 101, 65533, <---- suspicious! >> 1, 12, 10, >> 10, 109, 111, >> 110, 107, 101, >> 121, 98, 117, >> 116, 116, [length]: 26 >> ]] >> Sanity check error: Could not decode msg { >> decodeExc: Error: invalid wire type 7 at offset 18 >> at Reader.skipType (node_modules/protobufjs/src/reader.js:377:19) >> at Type.TestResponse$decode [as decode] (eval at Codegen >> (node_modules/@protobufjs/codegen/index.js:50:33), <anonymous>:20:5) >> at buildEncodedStr (test/utils/amqprototest.js:94:34) >> at Timeout._onTimeout (test/utils/amqprototest.js:57:17) >> at listOnTimeout (internal/timers.js:531:17) >> at processTimers (internal/timers.js:475:7) >> } >> >> On Friday, January 8, 2021 at 6:25:41 AM UTC-7 peo stri tise safe wrote: >> >>> Gotcha, so the index value of 100 is not a non-standard value. >>> >>> So I'm seeing "invalid wire type" when using the decode method of >>> protobuf.js of an AMQ body. This is in a Node using the protobuf.js, >>> stomp-client.js and an AMQ broker. >>> >>> The GPB that I'm using has a basic format of this: >>> >>> syntax = "proto2"; >>> package testt; >>> message Monkey >>> { >>> optional string timestamp = 1; >>> } >>> message TestResponse >>> { >>> optional Monkey header = 100; >>> optional string mykey = 2; >>> optional string myvalue = 3; >>> } >>> >>> And the message is encoded and sent to the broker like this: >>> >>> // note TestRespGPBDef is created earlier >>> let payload = { >>> header: { >>> timestamp: 'monkeybutt' >>> }, >>> mykey: 'hi', >>> myvalue: 'there' >>> }; >>> let verifyErr = TestRespGPBDef.verify(payload); >>> if( verifyErr ) { >>> console.log(`GPB OBJECT/PAYLOAD ERROR`, {verifyErr}); >>> return; >>> } >>> let gmsg = TestRespGPBDef.create(payload); >>> let buff = TestRespGPBDef.encode(gmsg).finish(); >>> let str = buff.toString(); >>> stompClient.publish('mytopic',str); >>> >>> >>> And the message is rd'x and decoded by the consumer like this: >>> >>> // note TestRespGPBDef is created earlier >>> client.subscribe('mytopic', function(body, headers) { >>> utils.printObj(`CONSUMER << RXD AMQ header and body`); >>> utils.printObj(`AMQ body\n [${body.length}] elements`, body); >>> let binArray = utils.stringToBinArray(body); >>> utils.printObj(`As binArray`,binArray); >>> try { >>> let gpbMsg = TestRespGPBDef.decode(binArray); >>> utils.printObj('',gpbMsg); >>> } catch (decodeExc) { >>> console.log(`Could not decode msg`,{decodeExc}); >>> } >>> }); >>> >>> >>> When the consumer receives the body from AMQ and tries to decode it, >>> this is the exception: >>> >>> CONSUMER << RXD AMQ header and body >>> AMQ body: >>> [26] elements (typeof [string]): >>> ['\u0012\u0002hi\u001a\u0005there�\u0006\f\n\nmonkeybutt'] >>> As binArray (typeof [object]): [[ >>> 18, 2, 104, >>> 105, 26, 5, >>> 116, 104, 101, >>> 114, 101, 65533, >>> 6, 12, 10, >>> 10, 109, 111, >>> 110, 107, 101, >>> 121, 98, 117, >>> 116, 116, [length]: 26 >>> ]] >>> Could not decode msg { >>> decodeExc: Error: invalid wire type 7 at offset 18 >>> at Reader.skipType >>> (/mytest/node_modules/protobufjs/src/reader.js:377:19) >>> at Type.TestResponse$decode [as decode] (eval at Codegen >>> (/mytest/node_modules/@protobufjs/codegen/index.js:50:33), <anonymous>:20:5) >>> at >>> /Users/rcox/gitprojects/POSTAPOCALYPSE/speb-hmi/deviceModules/keplerAdapter/test/utils/amqprototest.js:37:39 >>> at >>> /Users/rcox/gitprojects/POSTAPOCALYPSE/speb-hmi/deviceModules/keplerAdapter/node_modules/stomp-client/lib/client.js:206:9 >>> at Array.map (<anonymous>) >>> at StompFrameEmitter.<anonymous> >>> (/mytest/node_modules/stomp-client/lib/client.js:205:28) >>> at StompFrameEmitter.emit (events.js:210:5) >>> at StompFrameEmitter.parseBody >>> (/mytest/node_modules/stomp-client/lib/parser.js:136:12) >>> at StompFrameEmitter.handleData >>> (/mytest/node_modules/stomp-client/lib/parser.js:42:12) >>> at Socket.<anonymous> >>> (/mytest/node_modules/stomp-client/lib/client.js:188:18) >>> } >>> >>> When the "header" index is set to 1, and the experiment is repeated, >>> the decoding does not throw the exception. >>> On Friday, January 8, 2021 at 12:52:16 AM UTC-7 [email protected] >>> wrote: >>> >>>> 100 isn't "non-standard" as such, and shouldn't cause anything to fail. >>>> What exactly are you seeing? >>>> >>>> The valid range is 1-536870911, omitting 19000-19999 (and any reserved >>>> areas in your specific messages) smaller numbers are cheaper (fewer bytes) >>>> to encode, so are usually preferred - but: that's it. >>>> >>>> I'm guessing you're actually using some non-compliant code that is >>>> assuming field-headers are single bytes? That is true for very low field >>>> numbers (4 bytes, so: 1-15), but field 100 will take two bytes for the >>>> header. Is that the problem here? >>>> >>>> On Thu, 7 Jan 2021, 22:00 peo stri tise safe, <[email protected]> >>>> wrote: >>>> >>>>> What is the purpose of setting starting index values for a GPB message >>>>> definition to something other than 1? I am seeing a value of 100 in a >>>>> particular application and it is causing the encoding of the message to >>>>> fail. Just wondered why the use of values other than 1 for the first >>>>> item >>>>> in the message. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/protobuf/957aae3b-32f8-4d9d-8f5f-732e26e08cedn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/protobuf/957aae3b-32f8-4d9d-8f5f-732e26e08cedn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/protobuf/2a726cea-c829-4527-9878-232f746b32b8n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/protobuf/2a726cea-c829-4527-9878-232f746b32b8n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Regards, > > Marc > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/6bca4f04-484b-4a35-9dab-2ade0f483d6bn%40googlegroups.com.
