Hi GZ, (const void *)tbuffer->buf seems problematic: it's a GByteArray*. tbuffer->buf->data should point to the data.
On Sun, Dec 6, 2015 at 7:10 AM GZ <[email protected]> wrote: > Hi Nobuaki, > > That was my understanding too. So when I write the object I would expect > to see the data serialised into the memory buffer... in tbuffer->buf But I > dont. It would be hard to miss the long "XXXXX.." string. > > The data must be correctly stored because I was able load the data into a > new object correctly. > > I did try your suggestion and try deserialise the buffer after it had been > moved but this did not work, the object was empty. > > Its very frustrating. I tried the same process using the cpp library and > it works! > > 013394F0 | 08000100 00000206 0002012C 0A000300 | ...........,.... > 01339500 | 00000000 63BF280A 00040000 00000063 | ....c.(........c > 01339510 | C0F20A00 05000000 000063BF BE0B0006 | ..........c..... > 01339520 | 0000001D 58585858 58585858 58585858 | ....XXXXXXXXXXXX > 01339530 | 58585858 58585858 58585858 58585858 | XXXXXXXXXXXXXXXX > 01339540 | 5800 | X. > > I think I must be getting the buffer incorrectly because the output above > is completely different for the buffer I get in the c_glib version > (identical object). > > Any Ideas ?! > > Thanks again > George > > > > Nobuaki Sukegawa <[email protected]> wrote .. > > Hi GZ, > > > > > string laid out as if it was going out "over-the-wire" > > That's the whole point of memory buffer and it is supposed to work for > your > > use case. > > Even with the correct length you don't see the string ? > > You may want to just try sending the data of length reported by *read* > > method and deserializing on the other side of the wire. > > > > On Tue, Dec 1, 2015 at 4:00 AM GZ <[email protected]> wrote: > > > > > Hi Nobuaki, > > > > > > Thanks... I will keep an eye out for the fix. As you pointed out, using > > > read returns the correct length (as far as I can tell) and I am able to > > > deserialise all the data into a new object correctly. Thanks! However, > when > > > I look at the memory buffer, I still do not see the string. I suspect > that > > > the buffer contains a pointer to the string, not the string laid out > as if > > > it was going out "over-the-wire". > > > > > > Mayber I am accessing the buffer incorrectly or using the wrong > transport? > > > Ultimately I want to be able to take the whole buffer and send it to > Kafka > > > as a message for desrialisation by other components. > > > > > > Your help is much appreciated! > > > > > > George > > > > > > > > > > > > > > > Nobuaki Sukegawa <[email protected]> wrote .. > > > > Hi GZ, > > > > > > > > You're absolutely right about both "&" and the length/contents. > > > > It turned out that it is a bug: as you pointed out the write method > > > ignores > > > > some bytes written. > > > > So the data is written and ready to be deserialized but the reported > > > length > > > > is wrong. > > > > If you happen to be building Thrift from the source, the fix will be > > > > available on master branch soon. > > > > > > > > A very dirty and non-performant workaround is to deserialize the > buffer > > > > only to know the byte length before sending: > > > > wrong_len = cls->write(THRIFT_STRUCT(pData2), ...); > > > > correct_len = cls->read(THRIFT_STRUCT(pData2), ...); > > > > you'll see correct byte length since read is not affected by this > bug. > > > > > > > > On Sun, Nov 29, 2015 at 4:22 AM GZ <[email protected]> wrote: > > > > > > > > > Hi Nobuaki, > > > > > > > > > > I tried this originally but write is expecting a ThriftStruct* > rather > > > than > > > > > a ThriftStruct. I have tried with both &pData->parent and pData.. > the > > > > > results are pretty much the same. Below is a Hex printout of the > memory > > > > > buffer (19 bytes). > > > > > > > > > > &pData->parent > > > > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R....... > > > > > 01FB5050 | 010000 > > > > > > > > > > pData > > > > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R....... > > > > > 01EA4050 | 010000 > > > > > > > > > > > > > > > I would expect to see the "XXXXXXX" string serialised into this > buffer > > > but > > > > > it is not. > > > > > > > > > > George > > > > > > > > > > > > > > > > > > > > Nobuaki Sukegawa <[email protected]> wrote .. > > > > > > Hi GZ, > > > > > > > > > > > > Can you try with "pData->parent" part applied too ? > > > > > > Sorry I should have mentioned this too. > > > > > > > > > > > > > > > > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <[email protected]> wrote: > > > > > > > > > > > > > Hi Nobuaki, > > > > > > > > > > > > > > Yes - the rdata_write was in the generated c file not the > header. > > > > > > > > > > > > > > I made the changes you suggested, it was just what I was > looking > > > for. I > > > > > > > have repeated the corrected code to help others but > unfortunately > > > the > > > > > > > original problem persists.. > > > > > > > > > > > > > > > > > > > > > tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER, > "buf_size", > > > > > 1000, > > > > > > > NULL); > > > > > > > if(tbuffer){ > > > > > > > transport = THRIFT_TRANSPORT(tbuffer); > > > > > > > thrift_transport_open (transport, &error); > > > > > > > > > > > > > > protocol = THRIFT_PROTOCOL(g_object_new > > > > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL)); > > > > > > > if(protocol){ > > > > > > > rdata* pData; > > > > > > > pData = g_object_new (TYPE_RDATA,"flag",1, > > > > > > > "id",27, > > > > > > > "start",12000, > > > > > > > "done",345435, > > > > > > > "notif",34455, > > > > > > > > > > > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL); > > > > > > > if(pData){ > > > > > > > ThriftStructClass* cls = > > > > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData)); > > > > > > > len = cls->write(pData,protocol,&error); > > > > > > > > > > > > > > printf ("%d Bytes written to buffer\n", len); > > > > > > > > > > > > > > if(tbuffer->buf != NULL){ > > > > > > > printHex((const void *)tbuffer->buf, 100); > > > > > > > } > > > > > > > g_object_unref (pData); > > > > > > > } > > > > > > > g_object_unref (protocol); > > > > > > > } > > > > > > > g_object_unref(tbuffer); > > > > > > > } > > > > > > > > > > > > > > > > > > > > > The returned buffer length still appears incorrect and the > string > > > does > > > > > not > > > > > > > appear to be serialised into the buffer. > > > > > > > > > > > > > > I can track the writes through the writing function and I > notice > > > that > > > > > for > > > > > > > the spid string value > > > > > > > > > > > > > > thrift_protocol_write_field_begin returns 3 bytes > > > > > > > thrift_protocol_write_string returns 50 bytes > > > > > > > thrift_protocol_write_field_end returns 0 > > > > > > > > > > > > > > the 50 bytes never get added to the length because it is > overriden > > > by > > > > > the > > > > > > > 0. This is a snippit of the code.. > > > > > > > > > > > > > > if ((ret = thrift_protocol_write_field_begin (protocol, > "spid", > > > > > > > T_STRING, 6, error)) < 0) > > > > > > > return -1; > > > > > > > xfer += ret; > > > > > > > if ((ret = thrift_protocol_write_string (protocol, > > > this_object->spid, > > > > > > > error)) < 0) > > > > > > > return -1; > > > > > > > if ((ret = thrift_protocol_write_field_end (protocol, > error)) < > > > 0) > > > > > > > return -1; > > > > > > > xfer += ret; > > > > > > > > > > > > > > The result is that the write function returns 19 bytes and > when I > > > view > > > > > the > > > > > > > buffer I see no string. I guess I must still be missing > > > something... > > > > > The > > > > > > > spid string is 46 characters long. I would expect to see this > > > > > reflected in > > > > > > > the length of the buffer. > > > > > > > > > > > > > > The idl record is > > > > > > > > > > > > > > enum Status { > > > > > > > ONE = 1, > > > > > > > TWO = 2, > > > > > > > THREE = 4, > > > > > > > } > > > > > > > > > > > > > > struct rdata { > > > > > > > 1: Status flag, > > > > > > > 2: i16 id, > > > > > > > 3: i64 start, > > > > > > > 4: i64 done, > > > > > > > 5: i64 notif, > > > > > > > 6: string spid > > > > > > > } > > > > > > > > > > > > > > Any idea what might be happenning? > > > > > > > > > > > > > > Thanks again... > > > > > > > > > > > > > > George > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nobuaki Sukegawa <[email protected]> wrote .. > > > > > > > > It is strange, the generated header has rdata_write ? > > > > > > > > Seeing generated code, it should be something like this: > > > > > > > > > > > > > > > > ThriftStructClass* cls = > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData)); > > > > > > > > cls->write(pData->parent, protocol, &error); > > > > > > > > > > > > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <[email protected]> wrote: > > > > > > > > > > > > > > > > > I am new to thrift and have been trying to serialize a > thrift > > > > > structure > > > > > > > > > using c but have been unable to get it to work. I want to > > > serialize > > > > > > > using > > > > > > > > > the binary protocol into a memory buffer. I do not need > RPC, I > > > > > will be > > > > > > > > > passing the message into Kafka. Does anyone know the > correct > > > way > > > > > to do > > > > > > > this? > > > > > > > > > > > > > > > > > > I have looked through all the examples and test code and > have > > > come > > > > > up > > > > > > > with > > > > > > > > > the following. It compiles, but is not working correctly. > I am > > > > > clearly > > > > > > > not > > > > > > > > > doing it correctly. rdata_write is a method generated by > > > thrift. > > > > > > > > > > > > > > > > > > GError *error = NULL; > > > > > > > > > ThriftTransport *transport = NULL; > > > > > > > > > ThriftProtocol *protocol = NULL; > > > > > > > > > > > > > > > > > > transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER, > > > "buf_size", > > > > > 100, > > > > > > > > > NULL); > > > > > > > > > if(transport){ > > > > > > > > > thrift_transport_open (transport, &error); > > > > > > > > > protocol = g_object_new > (THRIFT_TYPE_BINARY_PROTOCOL, > > > > > > > "transport", > > > > > > > > > transport,NULL); > > > > > > > > > if(protocol){ > > > > > > > > > rdata* pData = g_object_new > (TYPE_RDATA,"flag",1, > > > > > > > > > "id",27, > > > > > > > > > "start",12000, > > > > > > > > > "done",345435, > > > > > > > > > "notif",34455, > > > > > > > > > "spid","This is the > > > > > SPID",NULL); > > > > > > > > > if(pData){ > > > > > > > > > > > > > > > > > > /* wanted something like > > > > > obj->write(protocol,&error) > > > > > > > > > But how to get object? pData does not > have > > > write > > > > > > > method > > > > > > > > > */ > > > > > > > > > > > > > > > > > > gint32 len = > > > > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error); > > > > > > > > > > > > > > > > > > // Print Content of Buffer > > > > > > > > > ThriftMemoryBuffer* pMem = > > > > > > > (ThriftMemoryBuffer*)transport; > > > > > > > > > if(pMem->buf != NULL){ > > > > > > > > > printHex((const void *)pMem->buf, len); > > > > > > > > > } > > > > > > > > > g_object_unref (pData); > > > > > > > > > } > > > > > > > > > g_object_unref (protocol); > > > > > > > > > } > > > > > > > > > g_object_unref(transport); > > > > > > > > > } > > > > > > > > > > > > > > > > > > Error handling and support functions skipped for brevity. > > > > > > > > > > > > > > > > > > Any help appreciated > > > > > > > > > > > > > > > > > > > > > > > > >
