Re: [protobuf] Dynamic/run-time decoding
Hi, due to organization changes and budget costs this task is put on hold. Thank you for all your guidance up until now. If any further help is needed regarding this issue in the future me or someone else will continue with this thread. On Thursday, 11 September 2014 20:00:52 UTC+2, Feng Xiao wrote: On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com javascript: wrote: Hi, Thank you that was one thing I tried but was not successful up until now. Also one more issue was that i had to ask for type by full name. It took me some time to figure it our. Now I got different sort of questions. Since I can now progress further I am finding new problems. As you suggested i will be using ParseFromString method. Although I am not entirely sure what this string is. Is it represented by hexdump or is it some other format? ParseFromString() accepts protobuf wire-format data (which is the output of protobuf serialization routines). I am also wondering about imports. If one proto file imports other do I have to take that consideration and manually handle that file as well or is it done automatically while processing file I have provided? In my implementation all proto files are in one folder with possible imports between each other. You can use the DescriptorDatabase implementation we use for protocol compiler: https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76 Basically rather than adding individual files to the pool, you provide a DescriptorDatabase to the pool and the pool will use this DescriptorDatabase to get the FileDescriptorProtos it needs. One culprit of this approach is that, you need to call DescriptorPool::FindFileByName() on the .proto files before you can use DescriptorPool::FindMessageTypeByName() to search for types. Thank you in advance for your response. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com javascript:. To post to this group, send email to prot...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi, due to organization changes and budget cuts this task is put on hold. Thank you for all your guidance up until now. If any further help is needed regarding this issue in the future me or someone else will continue with this thread. On Thursday, 11 September 2014 20:00:52 UTC+2, Feng Xiao wrote: On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com javascript: wrote: Hi, Thank you that was one thing I tried but was not successful up until now. Also one more issue was that i had to ask for type by full name. It took me some time to figure it our. Now I got different sort of questions. Since I can now progress further I am finding new problems. As you suggested i will be using ParseFromString method. Although I am not entirely sure what this string is. Is it represented by hexdump or is it some other format? ParseFromString() accepts protobuf wire-format data (which is the output of protobuf serialization routines). I am also wondering about imports. If one proto file imports other do I have to take that consideration and manually handle that file as well or is it done automatically while processing file I have provided? In my implementation all proto files are in one folder with possible imports between each other. You can use the DescriptorDatabase implementation we use for protocol compiler: https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76 Basically rather than adding individual files to the pool, you provide a DescriptorDatabase to the pool and the pool will use this DescriptorDatabase to get the FileDescriptorProtos it needs. One culprit of this approach is that, you need to call DescriptorPool::FindFileByName() on the .proto files before you can use DescriptorPool::FindMessageTypeByName() to search for types. Thank you in advance for your response. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com javascript:. To post to this group, send email to prot...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Null Nested Messages--whats expected?
I can't reproduce this outputting a 0x0; my example is below, and
outputs 22-0A-12-08-73-6F-6D-65-43-69-74-79 - the null nested message is
simply completely omitted; the contents are:
- 2 bytes field-header and length-prefix for the location member
- 2 bytes field-header and length-prefix for the city member
- 8 bytes payload for the text someCity
Happy to investigate, but a repro would really help. To clarify: it is
incorrect to emit an 0x0 at the point of a field-header; that is never
valid.
As a random shot-in-the-dark guess, is it possible that you're actually
using a `MemoryStream` with `GetBuffer` but without the `Length`? if so,
there will be trailing garbage (which will default to 0x0) in the buffer,
precisely because `GetBuffer` returns the *oversized* backing buffer. To
get a right-sized array from `MemoryStream`, use `ToArray` instead (but
note that this involves an extra allocation / copy).
Marc
using ProtoBuf;
using System;
using System.IO;
static class Program
{
static void Main()
{
var a = new A {
location = new Location {
city = someCity,
coordinate = null
}
};
var ms = new MemoryStream();
Serializer.Serialize(ms, a);
var hex = BitConverter.ToString(
ms.GetBuffer(), 0, (int)ms.Length);
Console.WriteLine(hex);
// outputs: 22-0A-12-08-73-6F-6D-65-43-69-74-79
}
}
[ProtoContract]
class A
{
[ProtoMember(2)]
public string someField { get; set; }
[ProtoMember(4)]
public Location location { get; set; }
}
[ProtoContract]
class Location
{
[ProtoMember(1)]
public Coordinate coordinate { get; set; }
[ProtoMember(2)]
public string city { get; set; }
}
[ProtoContract]
class Coordinate
{
//...
}
On 14 October 2014 18:55, Johannes Elgh johannes.e...@tink.se wrote:
Hello everyone,
Firstly I need to apologize for any misuse or misunderstand of terms and
concepts. I'm mostly familiar with the Protocol Buffer implementations
protostuff and protobuf-net.
I have a case that two different implementations (the C# one: protobuf-net
and the Java one: protostuff) do differently and I'd like to know which way
is the correct according to the Protocol Buffers specification.
I have these messages:
message A {
optional string someField = 2;
optional Location location = 4;
}
message Location {
optional Coordinate coordinate = 1;
optional string city = 2;
}
message Coordinate {
...
}
In the following case we have an object A, with a populated Location with
coordinate
= null and city = someCity.
When protobuf-net (a client) sends this object A (populated as described
above) to protostuff (the backend) it sends 0x0 in the place of the
coordinate.
This results into that protostuff reads a 0 whilst populating Location,
and therefore thinks that Location is done. When protostuff then reads
the next field number it finds 2, city, which has the same number as
someField in A and populates that field instead (city overwrites someField
).
So the problem is that protobuf-net sends 0x0 for a null nested messages
whereas protostuff doesn't even expect it to be there (since it's null).
Best regards,
Johannes
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
Regards,
Marc
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Null Nested Messages--whats expected?
Just for visibility; we've discussed this more on github, and it *looks* to be a protostuff decoding issue, not a protobuf-net encoding issue; Johannes will take our findings to protostuff for further investigation. Arbitrary link is arbitrary: https://github.com/mgravell/protobuf-net/issues/35 Marc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Null Nested Messages--whats expected?
For full visibility:
It turns out that the error was on my side. Apparently ProtoStuff can talk
different dialects. ProtoStuff and ProtoBuf.
It just depends on what IOUtil implementation you use.
When you read for instance, either use:
ProtobufIOUtil.mergeFrom(inputStream, t, schema, buffer);
or
ProtostuffIOUtil.mergeFrom(inputStream, t, schema, buffer);
This wasn't really clear from their side. I hade no clue. We had of course
been using ProtostuffIOUtil -- when we should have used ProtobufIOUtil
since we are talking between different implementations (ProtoStuff
protobuf-net).
Best,
Johannes
On Tuesday, October 14, 2014 7:55:58 PM UTC+2, Johannes Elgh wrote:
Hello everyone,
Firstly I need to apologize for any misuse or misunderstand of terms and
concepts. I'm mostly familiar with the Protocol Buffer implementations
protostuff and protobuf-net.
I have a case that two different implementations (the C# one: protobuf-net
and the Java one: protostuff) do differently and I'd like to know which way
is the correct according to the Protocol Buffers specification.
I have these messages:
message A {
optional string someField = 2;
optional Location location = 4;
}
message Location {
optional Coordinate coordinate = 1;
optional string city = 2;
}
message Coordinate {
...
}
In the following case we have an object A, with a populated Location with
coordinate
= null and city = someCity.
When protobuf-net (a client) sends this object A (populated as described
above) to protostuff (the backend) it sends 0x0 in the place of the
coordinate.
This results into that protostuff reads a 0 whilst populating Location,
and therefore thinks that Location is done. When protostuff then reads
the next field number it finds 2, city, which has the same number as
someField in A and populates that field instead (city overwrites someField
).
So the problem is that protobuf-net sends 0x0 for a null nested messages
whereas protostuff doesn't even expect it to be there (since it's null).
Best regards,
Johannes
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Message encoding question
Hello, I am quite new to Protocol Buffers but I have read the tutorials. There is one part of the encoding which I don't understand. How the message types are encoded? As I see the tutorial mentions the field encoding but the field id is unique only in the message. Best regards, Tamas -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Message encoding question
Hello, I am quite new to Protocol Buffers but I have read the tutorials. There is one part of the encoding which I don't understand. How the message types are encoded? As I see the tutorial mentions the field encoding but the field id is unique only in the message. Best regards, Tamas -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Message encoding question
On Wed, Oct 15, 2014 at 6:05 AM, Tamás Somhegyi somhegyi.ta...@gmail.com wrote: Hello, I am quite new to Protocol Buffers but I have read the tutorials. There is one part of the encoding which I don't understand. How the message types are encoded? As I see the tutorial mentions the field encoding but the field id is unique only in the message. They're not. It's up to you to know what message you're decoding. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 447 in protobuf: No C++11 move constructors
Updates: Status: WontFix Owner: xiaof...@google.com Comment #10 on issue 447 by xiaof...@google.com: No C++11 move constructors https://code.google.com/p/protobuf/issues/detail?id=447 FYI. Google's Style Guide now permits rvalue references to be used in move constructors and this feature for protobuf has been proposed and discussed again (google internally). It's rejected again unfortunately. For Googlers who may want a reference, see CL 67127736. Basically he who has to power to decide whether to accept this change is not convinced that it can improve the API or can improve the performance. Unless the issue is resolved internally, we wouldn't add move constructors to protobuf. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 447 in protobuf: No C++11 move constructors
Comment #11 on issue 447 by chen3feng: No C++11 move constructors https://code.google.com/p/protobuf/issues/detail?id=447 Very disappointed to hear this message. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 447 in protobuf: No C++11 move constructors
Comment #12 on issue 447 by cbsm...@gmail.com: No C++11 move constructors https://code.google.com/p/protobuf/issues/detail?id=447 Speaks volumes that after 22 months after submission and 5 months after a working plugin was contributed, there isn't even a direct answer. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 447 in protobuf: No C++11 move constructors
Comment #13 on issue 447 by chen3feng: No C++11 move constructors https://code.google.com/p/protobuf/issues/detail?id=447 let't make a fork on github? 2014-10-16 上午11:01于 proto...@googlecode.com写道： -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
