Hello Marc,
Thanks for providing protobuf-net to community.
I am facing same error message with Protobuf-net version 3.0.29. Error
message is "Invalid wire-type (Varint); this usually means you have
over-written a file without truncating or setting the length"
I thought main reason is Encoding type in my case. I passed the Hex string
and having Encoding type = RawEncoding then its work good. But with other
data with Encoding Type = HeatshrinkLzssEncoding, it gives me above
mentioned error. Should it not support this encoding type?
Thanks in advance.
On Thursday, December 24, 2009 at 3:23:32 AM UTC+5:30, Marc Gravell wrote:
>
> OK; now it makes sense. With the description and sample code above, this
> is nothing to do with protobuf-net (specifically), and everything to do
> with how you are doing your file access. If you use File.Open with
> OpenOrCreate on an **existing** file, it doesn't truncate the file. If you
> then write *fewer* bytes than were in the original file, you essentially
> have garbage at the end of the stream.
>
> The "protocol buffers" wire format doesn't (by default) include a length
> prefix for the overall message, since it is designed to be possible to
> merge by appending. You have a few options:
>
> 1: delete the file before re-writing it (although this may impact any ACLs
> etc that you have defined)
> 2: specify FileMode.Truncate rather than FileMode.OpenOrCreate (truncates
> old data *before* writing)
> 3: call stream.SetLength(stream.Position) after writing
> (truncates old data *after* writing)
> 4: use SerializeWithLengthPrefix (but note that this will *retain* old
> garbage)
>
> To reproduce just the stream aspects here, see below.
>
> Marc
>
> using System;
> using System.IO;
>
> static class Program {
> static Random rand = new Random();
> const string Path = @"scratch.bin";
> static void WriteGarbabe(Stream stream, int length) {
> byte[] data = new byte[length];
> rand.NextBytes(data);
> stream.Write(data, 0, length);
> }
> static void ShowLength(string path) {
> Console.WriteLine(new FileInfo(path).Length);
> }
> static void Main() {
> using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
> WriteGarbabe(dest, 50);
> }
> ShowLength(Path); // shows 50 like we expect
> using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
> WriteGarbabe(dest, 40);
> }
> ShowLength(Path); // shows 50, we might have expected 40
> using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
> WriteGarbabe(dest, 30);
> dest.SetLength(dest.Position);
> }
> ShowLength(Path); // shows 30; SetLength works
> using (Stream dest = File.Open(Path, FileMode.Truncate)) {
> WriteGarbabe(dest, 20);
> dest.SetLength(dest.Position);
> }
> ShowLength(Path); // shows 20; Truncate works
> }
> }
>
> 2009/12/22 jeevankodali <jeevan...@gmail.com <javascript:>>
>
>> Finally I was able to reproduce the cause of this error. I am not
>> getting the same error in this sample code but a different one (but I
>> think reason is the same).
>>
>> This is what I did which caused this:
>>
>> 1. First I stored a dictionary, key is int and value is a list into
>> the proto file and loaded this data - this step was fine.
>> 2. Next I changed my code to store just list instead of dictionary in
>> the same file - store was successful
>> 3. I tried to load this new file (which was the same name and location
>> as the original file with dictionary), I was getting the error.
>>
>> But when I deleted the original file and reran the code (which was
>> storing the list), everything was fine.
>>
>> Here is the code:
>>
>> [ProtoContract]
>> public class ProtoTestClass
>> {
>> private string _id;
>>
>> [ProtoMember(1)]
>> public string Id
>> {
>> get { return _id; }
>> set { _id = value; }
>> }
>> }
>>
>> // below code throws error because I am using the same file
>> private static void ProtoTest()
>> {
>> List<ProtoTestClass> ownerList = new List<ProtoTestClass>
>> ();
>> ProtoTestClass owner = new ProtoTestClass();
>> owner.Id = "799";
>> ownerList.Add(owner);
>>
>> Dictionary<int, List<ProtoTestClass>> dictionaryList = new
>> Dictionary<int, List<ProtoTestClass>>();
>> dictionaryList.Add(1, ownerList);
>>
>> using (Stream outfile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> ProtoBuf.Serializer.Serialize(outfile,
>> dictionaryList);
>> }
>>
>> Dictionary<int, List<ProtoTestClass>> outDictionaryList =
>> new Dictionary<int, List<ProtoTestClass>>();
>> using (Stream inFile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> outDictionaryList =
>> ProtoBuf.Serializer.Deserialize<Dictionary<int, List<ProtoTestClass>>>
>> (inFile);
>> }
>>
>>
>> // now write just list again in the same file
>> using (Stream outfile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> ProtoBuf.Serializer.Serialize(outfile, ownerList);
>> }
>>
>> List<ProtoTestClass> outList = new List<ProtoTestClass>();
>> using (Stream inFile = File.Open(@"d:\protobuftest.bin",
>> FileMode.OpenOrCreate))
>> {
>> outList =
>> ProtoBuf.Serializer.Deserialize<List<ProtoTestClass>>(inFile);
>> }
>>
>> Console.WriteLine("Done with test");
>> }
>>
>>
>> On Dec 22, 5:40 am, Marc Gravell <marc.grav...@gmail.com> wrote:
>> > Really, it is whatever is enough to show the problem. For example, the
>> > following doesn't (for me) show any issues. You might also want to
>> clarify
>> > your setup - could it be some x64/ia64 related bug, for example? (I'm
>> > testing on x86, Win7):
>> >
>> > using System.Collections.Generic;
>> > using System.Diagnostics;
>> > using System.IO;
>> > using ProtoBuf;
>> >
>> > [ProtoContract]
>> > public class ProtoBlock {
>> > private string date;
>> > [ProtoMember(1)]
>> > public string Date {
>> > get { return date; }
>> > set { date = value; }
>> > }
>> >
>> > }
>> >
>> > static class Program {
>> > static void Main() {
>> > List<ProtoBlock> data = new List<ProtoBlock>();
>> > RoundTrip(data); // 0
>> > data.Add(new ProtoBlock { Date = "abc" });
>> > RoundTrip(data); // 1
>> > data.Add(new ProtoBlock { Date = "def" });
>> > RoundTrip(data); // 2
>> > data.Add(new ProtoBlock { Date = "ghi" });
>> > RoundTrip(data); // 3
>> > }
>> > static void RoundTrip(List<ProtoBlock> list) {
>> > string path = "foo.bin";
>> > if (File.Exists(path)) File.Delete(path);
>> > using (Stream outfile = File.Open(path, FileMode.OpenOrCreate))
>> {
>> > ProtoBuf.Serializer.Serialize(outfile, list);
>> > }
>> > List<ProtoBlock> clone;
>> > using (Stream inFile = File.OpenRead(path)) {
>> > clone =
>> > ProtoBuf.Serializer.Deserialize<List<ProtoBlock>>(inFile)
>> > ?? new List<ProtoBlock>(); // blanks become null -
>> > limitation
>> > }
>> > Debug.Assert(!ReferenceEquals(clone, list)); // different lists
>> > Debug.Assert(clone.Count == list.Count); // same count
>> > for (int i = 0; i < list.Count; i++) {
>> > Debug.Assert(clone[i].Date == list[i].Date); // same data
>> > }
>> > }
>> >
>> > }
>> >
>> > 2009/12/21 jeevankodali <jeevankod...@gmail.com>
>> >
>> >
>> >
>> >
>> >
>> > > Sorry for the late reply, what exactly do you want in the full
>> > > example, I can add the code that I am using, do you need the file as
>> > > well (and how can I send the file)?
>> >
>> > > Thanks
>> >
>> > > On Dec 19, 5:09 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
>> > > > Rechecked with Net20 from the r278 build, and it worked fine. Sorry
>> to be
>> > > a
>> > > > pain, but do you have a complete example that shows the issue?
>> >
>> > > > Marc
>> >
>> > > > 2009/12/19 Marc Gravell <marc.grav...@gmail.com>
>> >
>> > > > > OK; I'll re-check with the Net20 variant...
>> >
>> > > > > 2009/12/19 jeevankodali <jeevankod...@gmail.com>
>> >
>> > > > > Thanks for the reply. I used r278 and Net20 folder in that zip
>> file.
>> >
>> > > > >> On Dec 18, 2:36 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
>> > > > >> > Hmmm... that should work fine; there *was* a related bug in an
>> early
>> > > > >> build,
>> > > > >> > but...
>> >
>> > > > >> > Can I check which version and framework you are using? I've
>> checked
>> > > on
>> > > > >> r275,
>> > > > >> > and it works fine "as is", for both empty lists and some
>> arbitrary
>> > > data
>> > > > >> I
>> > > > >> > made up.
>> >
>> > > > >> > Sorry for the delay, btw - seasonal break, etc...
>> >
>> > > > >> > Marc Gravell (protobuf-net)
>> >
>> > > > >> --
>> >
>> > > > >> You received this message because you are subscribed to the
>> Google
>> > > Groups
>> > > > >> "Protocol Buffers" group.
>> > > > >> To post to this group, send email to prot...@googlegroups.com
>> <javascript:>.
>> > > > >> To unsubscribe from this group, send email to
>> > > > >> prot...@googlegroups.com <javascript:>
>> <protobuf%2Bunsubscribe@googlegroups.com>
>> > > <protobuf%2Bunsubscr...@googlegroups.com>
>> > > > >> .
>> > > > >> For more options, visit this group at
>> > > > >>http://groups.google.com/group/protobuf?hl=en.
>> >
>> > > > > --
>> > > > > Regards,
>> >
>> > > > > Marc
>> >
>> > > > --
>> > > > Regards,
>> >
>> > > > Marc- Hide quoted text -
>> >
>> > > > - Show quoted text -
>> >
>> > > --
>> >
>> > > You received this message because you are subscribed to the Google
>> Groups
>> > > "Protocol Buffers" group.
>> > > To post to this group, send email to prot...@googlegroups.com
>> <javascript:>.
>> > > To unsubscribe from this group, send email to
>> > > prot...@googlegroups.com <javascript:>
>> <protobuf%2Bunsubscribe@googlegroups.com>
>> > > .
>> > > For more options, visit this group at
>> > >http://groups.google.com/group/protobuf?hl=en.
>> >
>> > --
>> > Regards,
>> >
>> > Marc- Hide quoted text -
>> >
>> > - Show quoted text -
>>
>> --
>>
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to prot...@googlegroups.com
>> <javascript:>.
>> To unsubscribe from this group, send email to prot...@googlegroups.com
>> <javascript:>.
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>>
>
>
> --
> Regards,
>
> Marc
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/5b8ff9b8-2979-458e-80ce-bef874117f73o%40googlegroups.com.
