Re: Protobuf-net Serialize problem
Marc,
another issue is that for optional string fields, protogen defaults to
"". Shouldn't it default to null?
optional string clientid = 2;
private string _clientid = "";
[ProtoBuf.ProtoMember(2, IsRequired = false, Name =
@"clientid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string clientid
{
get { return _clientid; }
set { _clientid = value; }
}
Thanks
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Marc,
ACtually, not using detectMissing will fix the string serialization
but not the primitive types which would default to default(T). How
will protobuf-net serialize the field if I munged the signature as
public int? Foo { get; set; }
Thanks
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Marc,
We figured out the problem. I had to send it over TIBCO as byte[] and
on the server side, it had to be pulled out as tibrv_u8array. After
that, the protobuf string conversion worked.
However, I noticed couple of odd things in this latest release of
protobuf-net.
1) It seems that an Optional Int field generated with detectMissing
flag does not get serialized if it was set to 0 (meaning *Specified
field was true but the field didn't reach the server).
2) optional string fields are being handled incorrectly.
public string execid
{
get { return _execid ?? ""; }
set { _execid = value; }
}
If execId was null, it will get serialized as an empty string which
means it ends up getting on the wire.
I'm going to revert back to gen'ing the class without the
detectMissing flag.
Thanks
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
I'm not an expect on TIBCO messages; and I'm not sure what AddField
(string,byte[]) method you are talking about, sorry.
But if that is a Tibco method for packing a (named) byte[] into a
Tibco message, then it sounds ideal. Serialize to a MemoryStream and
use .ToArray() to get the byte[] - for example:
using(var ms = new MemoryStream()) {
Serializer.Serialize(ms, obj);
message.AddField("pb", ms.ToArray());
}
This is based purely on speculation about the AddField message! But it
sounds promising.
Marc
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Marc, Yes, we suspect the problem is in the transmit from .Net as well due to the truncation at 1st NULL char in the Unicode string. I'm sending the data back as a TIBCO message which also adds the field as a string. Do you think if I use the AddField(string, byte[]) overload, it'll work on the server? Thanks On Apr 4, 6:04 am, Marc Gravell wrote: > I've just been looking more - and ParseFromString should do the job. > I /suspect/ the problem is the definition a "string"... you aren't > going to be able to treat raw binary in .NET as a String; you are > going to have to treat it as a Stream or a byte[], or you can encode > it to base-64 (as above) to get something that *can* be treated as a > String. > > Either as binary or a base-64, you would then transfer that to the > server. At the server, ProtoBuf again expects raw binary, but in a C++ > string (if you have used base-64 as a transfer mechanism, you would > have to decode the base-64 back to binary). You should then be able to > mount that data into a C++ string (or an istream). > > My best guess is that at the moment you are losing data during > transfer (as above). > > What mechanism are you using to transfer the data? If it is a network > socket (for example) you should be able to just send binary, without > ever touching a .NET string; likewise HTTP can be used without > creating a .NET string. > > Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
I've just been looking more - and ParseFromString should do the job. I /suspect/ the problem is the definition a "string"... you aren't going to be able to treat raw binary in .NET as a String; you are going to have to treat it as a Stream or a byte[], or you can encode it to base-64 (as above) to get something that *can* be treated as a String. Either as binary or a base-64, you would then transfer that to the server. At the server, ProtoBuf again expects raw binary, but in a C++ string (if you have used base-64 as a transfer mechanism, you would have to decode the base-64 back to binary). You should then be able to mount that data into a C++ string (or an istream). My best guess is that at the moment you are losing data during transfer (as above). What mechanism are you using to transfer the data? If it is a network socket (for example) you should be able to just send binary, without ever touching a .NET string; likewise HTTP can be used without creating a .NET string. Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
OK; can I clafy then; the server is using the standard C++ version of ProtoBuf 2.0.1, and the client is protobuf-net (recent version), and you are having difficulty using ParseFromString in the C++ version. Is that correct? I will need to check (Kenton?) what format ParseFromString is expecting, as I simply haven't had need to use this. I will launch a new thread to try to find out... Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Marc, The server is using ProtoBuf 2.0.1. I had to upgrade my .net version to the latest protobuf-net in order to have protogen work with optional fields correctly. It seems that the server only processes bytes up to the 1st NULL char when calling ParseFromString. Will this problem be avoided with Base64? Thanks On Apr 2, 5:52 pm, Marc Gravell wrote: > (re last post) > > As with unicode, you can't use ASCII to handle raw binary; either use > base-64 (previous post), or send binary. > > Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
(re last post) As with unicode, you can't use ASCII to handle raw binary; either use base-64 (previous post), or send binary. Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
You can't use unicode to transport arbitrary binary data - it won't be
a valid unicode string. If you have to use strings, then base-64 is
the safest option:
private string Serialize(OrderProto proto)
{
using (MemoryStream stream = new MemoryStream())
{
Serializer.Serialize(stream, proto);
return Convert.ToBase64String(stream.GetBuffer(), 0,
(int)stream.Length);
}
}
private OrderProto Deserialize(string str)
{
byte[] data = Convert.FromBase64String(str);
using (MemoryStream stream = new MemoryStream(data)) {
return Serializer.Deserialize(stream);
}
}
However, if possible, raw binary would be even more efficient.
I've tried the above (with the current build), and it round-trips
fine, giving the same length when repeated.
Does the above fix it at all?
Marc
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Quick question - which build are you using? (at client and server)? Based on the field names, the code-generation dates back to November; if the dlls are aged similarly, it is possible that this is an old bug. If possible, can you retry with the current build? I'm just trying the code you posted, to see if I can spot anything... Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Well, this code certainly passes the unit test of comparing before and
after protobufs. But the string still fails to be recognized as a
valid protobuf object on the server.
Any ideas?
Thanks
private string Serialize(OrderProto proto)
{
using (MemoryStream stream = new MemoryStream())
{
Serializer.Serialize(stream, proto);
byte[] data = stream.ToArray();
char[] chars = new char[stream.Length];
for (int i = 0; i < data.Length; i++)
chars[i] = (char) data[i];
return new string(chars);
}
}
private OrderProto Deserialize(string protoStr)
{
char[] chars = protoStr.ToCharArray();
byte[] data = new byte[chars.Length];
for (int i = 0; i < chars.Length; i++)
data[i] = (byte)chars[i];
using (MemoryStream stream = new MemoryStream(data))
return Serializer.Deserialize(stream);
}
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Actually, I was wrong, this code produces an empty string. The
following code serializes *something* which fails on the server side
on ParseFromString. Any ideas what I have to do to correctly serialize
this?
private string Serialize(OrderProto proto)
{
using (MemoryStream stream = new MemoryStream())
{
Serializer.Serialize(stream, proto);
stream.Seek(0, SeekOrigin.Begin);
using (BinaryReader reader = new BinaryReader(stream,
Encoding.ASCII))
{
char[] chars = new char[stream.Length];
for (int i = 0; i < stream.Length; i++)
chars[i] = reader.ReadChar();
return new string(chars);
}
}
}
Thanks
On Apr 2, 1:04 pm, "test.f...@nomail.please"
wrote:
> It seems I was wrong in trying to use ASCII or UTF-8 encoding. This
> works.
>
> private string Serialize(OrderProto proto)
> {
> using (MemoryStream stream = new MemoryStream())
> {
> Serializer.Serialize(stream, proto);
> using (StreamReader reader = new StreamReader(stream))
> return reader.ReadToEnd();
> }
> }
>
> Thanks
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
It seems I was wrong in trying to use ASCII or UTF-8 encoding. This
works.
private string Serialize(OrderProto proto)
{
using (MemoryStream stream = new MemoryStream())
{
Serializer.Serialize(stream, proto);
using (StreamReader reader = new StreamReader(stream))
return reader.ReadToEnd();
}
}
Thanks
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protobuf-net Serialize problem
Marc,
Thanks for responding.
As I'd mentioned, the server is expecting the proto serialized as
binary and passed as a string param. It's doing the following.
Order::ProtoBufType sourceBuff;
if(!sourceBuff.ParseFromString(params["Order"]->string_value())) {
// exception
I'm able to gen the proto and use it elsewhere successfully without
having to handle encoding. This is how the server sends out the msg.
e->order.SerializeToString(&sOrder);
This is my first attempt to *send* a proto back to the server. I
tried encoding it as Unicode and while I didn't get the "invalid wire
type" error, the package was still rejected by the server.
private string Serialize(OrderProto proto)
{
using (MemoryStream stream = new MemoryStream())
{
Serializer.Serialize(stream, proto);
byte[] data = new byte[stream.Length];
stream.Seek(0, SeekOrigin.Begin);
stream.Read(data, 0, data.Length);
return Encoding.Unicode.GetString(data);
}
}
private OrderProto TestDeserialize(string str)
{
byte[] data = Encoding.Unicode.GetBytes(str);
MemoryStream stream = new MemoryStream(data);
OrderProto proto = Serializer.Deserialize
(stream);
return proto;
}
Also, do you know why buffer.Length would be slightly smaller than
contentBytes.Length in the following:
byte[] contentBytes = contents.Value;
OrderProto order = Serializer.Deserialize(new MemoryStream
(contentBytes));
MemoryStream ms = new MemoryStream();
Serializer.Serialize(ms, order);
ms.Seek(0, SeekOrigin.Begin);
byte[] buffer = ms.ToArray();
Here's the OrderProto class.
Thanks
[System.Serializable, ProtoBuf.ProtoContract(Name = @"Order")]
public partial class OrderProto : ProtoBuf.IExtensible
{
public OrderProto() { }
private string _ID0EW = "";
[ProtoBuf.ProtoMember(1, IsRequired = false, Name =
@"clearingid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string clearingid
{
get { return _ID0EW; }
set { _ID0EW = value; }
}
private string _ID0E6 = "";
[ProtoBuf.ProtoMember(2, IsRequired = false, Name =
@"clientid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string clientid
{
get { return _ID0E6; }
set { _ID0E6 = value; }
}
private string _ID0EIB = "";
[ProtoBuf.ProtoMember(3, IsRequired = false, Name =
@"ordersource", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string ordersource
{
get { return _ID0EIB; }
set { _ID0EIB = value; }
}
private string _ID0ERB = "";
[ProtoBuf.ProtoMember(4, IsRequired = false, Name =
@"orderid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string orderid
{
get { return _ID0ERB; }
set { _ID0ERB = value; }
}
private ulong _ID0E1B = default(ulong);
[ProtoBuf.ProtoMember(5, IsRequired = false, Name =
@"ordercreationtime", DataFormat =
ProtoBuf.DataFormat.TwosComplement)]
[System.ComponentModel.DefaultValue(default(ulong))]
public ulong ordercreationtime
{
get { return _ID0E1B; }
set { _ID0E1B = value; }
}
private string _ID0EDC = "";
[ProtoBuf.ProtoMember(6, IsRequired = false, Name =
@"parentorderid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string parentorderid
{
get { return _ID0EDC; }
set { _ID0EDC = value; }
}
private string _ID0EMC = "";
[ProtoBuf.ProtoMember(7, IsRequired = false, Name =
@"rootorderid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string rootorderid
{
get { return _ID0EMC; }
set { _ID0EMC = value; }
}
private string _ID0EVC = "";
[ProtoBuf.ProtoMember(8, IsRequired = false, Name =
@"subsystemid", DataFormat = ProtoBuf.DataFormat.Default)]
[System.ComponentModel.DefaultValue("")]
public string subsystemid
{
get { return _ID0EVC; }
set { _ID0EVC = value; }
}
private ulong _ID0E5C = default(ulong);
[ProtoBuf.ProtoMember(9, IsRequired = false, Name = @"symbol",
DataFormat = ProtoBuf.DataFormat.TwosComplement)]
[System.ComponentModel.DefaultValue(default(ulong))]
public ulong symbol
{
get { return _ID0E5C; }
set { _ID0E5C = value; }
}
private readon
Re: Protobuf-net Serialize problem
Hi; I'm the author of protobuf-net, and I hope I can help you find what is going on here... The binary format should be compatible between platforms (that is the point, after all). Are you able to share your OrderProto class with me? Either here, or at marc.grav...@gmail.com. The base-64 should be unrelated as long as it is being packed/unpacked correctly. I know of many people using protobuf-net to communicate between both client/ server and windows/linux (mono), so it should work... If you are transporting over a network - then one common issue is that the protocol buffers format is not terminated; messages will bleed into each-other unless you separate messages. You can do this using SerializeWithLengthPrefix and DeserializeWithLengthPrefix, which include the length of the message and only read that much data from the stream. Marc Gravell --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
