Hello Marc,
Thanks for providing protobuf-net to community.
I am facing same error message with Protobuf-net version 3.0.29. Error
message is "Invalid wire-type (Varint); this usually means you have
over-written a file without truncating or setting the length"
I thought main reason is Encoding type in my case. I passed the Hex string
and having Encoding type = RawEncoding then its work good. But with other
data with Encoding Type = HeatshrinkLzssEncoding, it gives me above
mentioned error. Should it not support this encoding type?
Thanks in advance.
On Thursday, December 24, 2009 at 3:23:32 AM UTC+5:30, Marc Gravell wrote:
>
> OK; now it makes sense. With the description and sample code above, this
> is nothing to do with protobuf-net (specifically), and everything to do
> with how you are doing your file access. If you use File.Open with
> OpenOrCreate on an **existing** file, it doesn't truncate the file. If you
> then write *fewer* bytes than were in the original file, you essentially
> have garbage at the end of the stream.
>
> The "protocol buffers" wire format doesn't (by default) include a length
> prefix for the overall message, since it is designed to be possible to
> merge by appending. You have a few options:
>
> 1: delete the file before re-writing it (although this may impact any ACLs
> etc that you have defined)
> 2: specify FileMode.Truncate rather than FileMode.OpenOrCreate (truncates
> old data *before* writing)
> 3: call stream.SetLength(stream.Position) after writing
> (truncates old data *after* writing)
> 4: use SerializeWithLengthPrefix (but note that this will *retain* old
> garbage)
>
> To reproduce just the stream aspects here, see below.
>
> Marc
>
> using System;
> using System.IO;
>
> static class Program {
> static Random rand = new Random();
> const string Path = @"scratch.bin";
> static void WriteGarbabe(Stream stream, int length) {
> byte[] data = new byte[length];
> rand.NextBytes(data);
> stream.Write(data, 0, length);
> }
> static void ShowLength(string path) {
> Console.WriteLine(new FileInfo(path).Length);
> }
> static void Main() {
> using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
> WriteGarbabe(dest, 50);
> }
> ShowLength(Path); // shows 50 like we expect
> using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
> WriteGarbabe(dest, 40);
> }
> ShowLength(Path); // shows 50, we might have expected 40
> using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
> WriteGarbabe(dest, 30);
> dest.SetLength(dest.Position);
> }
> ShowLength(Path); // shows 30; SetLength works
> using (Stream dest = File.Open(Path, FileMode.Truncate)) {
> WriteGarbabe(dest, 20);
> dest.SetLength(dest.Position);
> }
> ShowLength(Path); // shows 20; Truncate works
> }
> }
>
> 2009/12/22 jeevankodali <[email protected] <javascript:>>
>
>> Finally I was able to reproduce the cause of this error. I am not
>> getting the same error in this sample code but a different one (but I
>> think reason is the same).
>>
>> This is what I did which caused this:
>>
>> 1. First I stored a dictionary, key is int and value is a list into
>> the proto file and loaded this data - this step was fine.
>> 2. Next I changed my code to store just list instead of dictionary in
>> the same file - store was successful
>> 3. I tried to load this new file (which was the same name and location
>> as the original file with dictionary), I was getting the error.
>>
>> But when I deleted the original file and reran the code (which was
>> storing the list), everything was fine.
>>
>> Here is the code:
>>
>> [ProtoContract]
>> public class ProtoTestClass
>> {
>> private string _id;
>>
>> [ProtoMember(1)]
>> public string Id
>> {
>> get { return _id; }
>> set { _id = value; }
>> }
>> }
>>
>> // below code throws error because I am using the same file
>> private static void ProtoTest()
>> {
>> List<ProtoTestClass> ownerList = new List<ProtoTestClass>
>> ();
>> ProtoTestClass owner = new ProtoTestClass();
>> owner.Id = "799";
>> ownerList.Add(owner);
>>
>> Dictionary<int, List<ProtoTestClass>> dictionaryList = new
>> Dictionary<int, List<ProtoTestClass>>();
>> dictionaryList.Add(1, ownerList);
>>
>> using (Stream outfile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> ProtoBuf.Serializer.Serialize(outfile,
>> dictionaryList);
>> }
>>
>> Dictionary<int, List<ProtoTestClass>> outDictionaryList =
>> new Dictionary<int, List<ProtoTestClass>>();
>> using (Stream inFile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> outDictionaryList =
>> ProtoBuf.Serializer.Deserialize<Dictionary<int, List<ProtoTestClass>>>
>> (inFile);
>> }
>>
>>
>> // now write just list again in the same file
>> using (Stream outfile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> ProtoBuf.Serializer.Serialize(outfile, ownerList);
>> }
>>
>> List<ProtoTestClass> outList = new List<ProtoTestClass>();
>> using (Stream inFile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> outList =
>> ProtoBuf.Serializer.Deserialize<List<ProtoTestClass>>(inFile);
>> }
>>
>> Console.WriteLine("Done with test");
>> }
>>
>>
>> On Dec 22, 5:40 am, Marc Gravell <[email protected]> wrote:
>> > Really, it is whatever is enough to show the problem. For example, the
>> > following doesn't (for me) show any issues. You might also want to
>> clarify
>> > your setup - could it be some x64/ia64 related bug, for example? (I'm
>> > testing on x86, Win7):
>> >
>> > using System.Collections.Generic;
>> > using System.Diagnostics;
>> > using System.IO;
>> > using ProtoBuf;
>> >
>> > [ProtoContract]
>> > public class ProtoBlock {
>> > private string date;
>> > [ProtoMember(1)]
>> > public string Date {
>> > get { return date; }
>> > set { date = value; }
>> > }
>> >
>> > }
>> >
>> > static class Program {
>> > static void Main() {
>> > List<ProtoBlock> data = new List<ProtoBlock>();
>> > RoundTrip(data); // 0
>> > data.Add(new ProtoBlock { Date = "abc" });
>> > RoundTrip(data); // 1
>> > data.Add(new ProtoBlock { Date = "def" });
>> > RoundTrip(data); // 2
>> > data.Add(new ProtoBlock { Date = "ghi" });
>> > RoundTrip(data); // 3
>> > }
>> > static void RoundTrip(List<ProtoBlock> list) {
>> > string path = "foo.bin";
>> > if (File.Exists(path)) File.Delete(path);
>> > using (Stream outfile = File.Open(path, FileMode.OpenOrCreate))
>> {
>> > ProtoBuf.Serializer.Serialize(outfile, list);
>> > }
>> > List<ProtoBlock> clone;
>> > using (Stream inFile = File.OpenRead(path)) {
>> > clone =
>> > ProtoBuf.Serializer.Deserialize<List<ProtoBlock>>(inFile)
>> > ?? new List<ProtoBlock>(); // blanks become null -
>> > limitation
>> > }
>> > Debug.Assert(!ReferenceEquals(clone, list)); // different lists
>> > Debug.Assert(clone.Count == list.Count); // same count
>> > for (int i = 0; i < list.Count; i++) {
>> > Debug.Assert(clone[i].Date == list[i].Date); // same data
>> > }
>> > }
>> >
>> > }
>> >
>> > 2009/12/21 jeevankodali <[email protected]>
>> >
>> >
>> >
>> >
>> >
>> > > Sorry for the late reply, what exactly do you want in the full
>> > > example, I can add the code that I am using, do you need the file as
>> > > well (and how can I send the file)?
>> >
>> > > Thanks
>> >
>> > > On Dec 19, 5:09 pm, Marc Gravell <[email protected]> wrote:
>> > > > Rechecked with Net20 from the r278 build, and it worked fine. Sorry
>> to be
>> > > a
>> > > > pain, but do you have a complete example that shows the issue?
>> >
>> > > > Marc
>> >
>> > > > 2009/12/19 Marc Gravell <[email protected]>
>> >
>> > > > > OK; I'll re-check with the Net20 variant...
>> >
>> > > > > 2009/12/19 jeevankodali <[email protected]>
>> >
>> > > > > Thanks for the reply. I used r278 and Net20 folder in that zip
>> file.
>> >
>> > > > >> On Dec 18, 2:36 pm, Marc Gravell <[email protected]> wrote:
>> > > > >> > Hmmm... that should work fine; there *was* a related bug in an
>> early
>> > > > >> build,
>> > > > >> > but...
>> >
>> > > > >> > Can I check which version and framework you are using? I've
>> checked
>> > > on
>> > > > >> r275,
>> > > > >> > and it works fine "as is", for both empty lists and some
>> arbitrary
>> > > data
>> > > > >> I
>> > > > >> > made up.
>> >
>> > > > >> > Sorry for the delay, btw - seasonal break, etc...
>> >
>> > > > >> > Marc Gravell (protobuf-net)
>> >
>> > > > >> --
>> >
>> > > > >> You received this message because you are subscribed to the
>> Google
>> > > Groups
>> > > > >> "Protocol Buffers" group.
>> > > > >> To post to this group, send email to [email protected]
>> <javascript:>.
>> > > > >> To unsubscribe from this group, send email to
>> > > > >> [email protected] <javascript:>
>> <protobuf%[email protected]om>
>> > > <protobuf%[email protected]om>
>> > > > >> .
>> > > > >> For more options, visit this group at
>> > > > >>http://groups.google.com/group/protobuf?hl=en.
>> >
>> > > > > --
>> > > > > Regards,
>> >
>> > > > > Marc
>> >
>> > > > --
>> > > > Regards,
>> >
>> > > > Marc- Hide quoted text -
>> >
>> > > > - Show quoted text -
>> >
>> > > --
>> >
>> > > You received this message because you are subscribed to the Google
>> Groups
>> > > "Protocol Buffers" group.
>> > > To post to this group, send email to [email protected]
>> <javascript:>.
>> > > To unsubscribe from this group, send email to
>> > > [email protected] <javascript:>
>> <protobuf%[email protected]om>
>> > > .
>> > > For more options, visit this group at
>> > >http://groups.google.com/group/protobuf?hl=en.
>> >
>> > --
>> > Regards,
>> >
>> > Marc- Hide quoted text -
>> >
>> > - Show quoted text -
>>
>> --
>>
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> To unsubscribe from this group, send email to [email protected]
>> <javascript:>.
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>>
>
>
> --
> Regards,
>
> Marc
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/5b8ff9b8-2979-458e-80ce-bef874117f73o%40googlegroups.com.
