OK; now it makes sense. With the description and sample code above, this is
nothing to do with protobuf-net (specifically), and everything to do with
how you are doing your file access. If you use File.Open with OpenOrCreate
on an **existing** file, it doesn't truncate the file. If you then write
*fewer* bytes than were in the original file, you essentially have garbage
at the end of the stream.
The "protocol buffers" wire format doesn't (by default) include a length
prefix for the overall message, since it is designed to be possible to merge
by appending. You have a few options:
1: delete the file before re-writing it (although this may impact any ACLs
etc that you have defined)
2: specify FileMode.Truncate rather than FileMode.OpenOrCreate (truncates
old data *before* writing)
3: call stream.SetLength(stream.Position) after writing (truncates old data
*after* writing)
4: use SerializeWithLengthPrefix (but note that this will *retain* old
garbage)
To reproduce just the stream aspects here, see below.
Marc
using System;
using System.IO;
static class Program {
static Random rand = new Random();
const string Path = @"scratch.bin";
static void WriteGarbabe(Stream stream, int length) {
byte[] data = new byte[length];
rand.NextBytes(data);
stream.Write(data, 0, length);
}
static void ShowLength(string path) {
Console.WriteLine(new FileInfo(path).Length);
}
static void Main() {
using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
WriteGarbabe(dest, 50);
}
ShowLength(Path); // shows 50 like we expect
using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
WriteGarbabe(dest, 40);
}
ShowLength(Path); // shows 50, we might have expected 40
using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {
WriteGarbabe(dest, 30);
dest.SetLength(dest.Position);
}
ShowLength(Path); // shows 30; SetLength works
using (Stream dest = File.Open(Path, FileMode.Truncate)) {
WriteGarbabe(dest, 20);
dest.SetLength(dest.Position);
}
ShowLength(Path); // shows 20; Truncate works
}
}
2009/12/22 jeevankodali <jeevankod...@gmail.com>
> Finally I was able to reproduce the cause of this error. I am not
> getting the same error in this sample code but a different one (but I
> think reason is the same).
>
> This is what I did which caused this:
>
> 1. First I stored a dictionary, key is int and value is a list into
> the proto file and loaded this data - this step was fine.
> 2. Next I changed my code to store just list instead of dictionary in
> the same file - store was successful
> 3. I tried to load this new file (which was the same name and location
> as the original file with dictionary), I was getting the error.
>
> But when I deleted the original file and reran the code (which was
> storing the list), everything was fine.
>
> Here is the code:
>
> [ProtoContract]
> public class ProtoTestClass
> {
> private string _id;
>
> [ProtoMember(1)]
> public string Id
> {
> get { return _id; }
> set { _id = value; }
> }
> }
>
> // below code throws error because I am using the same file
> private static void ProtoTest()
> {
> List<ProtoTestClass> ownerList = new List<ProtoTestClass>
> ();
> ProtoTestClass owner = new ProtoTestClass();
> owner.Id = "799";
> ownerList.Add(owner);
>
> Dictionary<int, List<ProtoTestClass>> dictionaryList = new
> Dictionary<int, List<ProtoTestClass>>();
> dictionaryList.Add(1, ownerList);
>
> using (Stream outfile = File.Open(@"d:\protobuftest.bin",
> FileMode.OpenOrCreate))
> {
> ProtoBuf.Serializer.Serialize(outfile,
> dictionaryList);
> }
>
> Dictionary<int, List<ProtoTestClass>> outDictionaryList =
> new Dictionary<int, List<ProtoTestClass>>();
> using (Stream inFile = File.Open(@"d:\protobuftest.bin",
> FileMode.OpenOrCreate))
> {
> outDictionaryList =
> ProtoBuf.Serializer.Deserialize<Dictionary<int, List<ProtoTestClass>>>
> (inFile);
> }
>
>
> // now write just list again in the same file
> using (Stream outfile = File.Open(@"d:\protobuftest.bin",
> FileMode.OpenOrCreate))
> {
> ProtoBuf.Serializer.Serialize(outfile, ownerList);
> }
>
> List<ProtoTestClass> outList = new List<ProtoTestClass>();
> using (Stream inFile = File.Open(@"d:\protobuftest.bin",
> FileMode.OpenOrCreate))
> {
> outList =
> ProtoBuf.Serializer.Deserialize<List<ProtoTestClass>>(inFile);
> }
>
> Console.WriteLine("Done with test");
> }
>
>
> On Dec 22, 5:40 am, Marc Gravell <marc.grav...@gmail.com> wrote:
> > Really, it is whatever is enough to show the problem. For example, the
> > following doesn't (for me) show any issues. You might also want to
> clarify
> > your setup - could it be some x64/ia64 related bug, for example? (I'm
> > testing on x86, Win7):
> >
> > using System.Collections.Generic;
> > using System.Diagnostics;
> > using System.IO;
> > using ProtoBuf;
> >
> > [ProtoContract]
> > public class ProtoBlock {
> > private string date;
> > [ProtoMember(1)]
> > public string Date {
> > get { return date; }
> > set { date = value; }
> > }
> >
> > }
> >
> > static class Program {
> > static void Main() {
> > List<ProtoBlock> data = new List<ProtoBlock>();
> > RoundTrip(data); // 0
> > data.Add(new ProtoBlock { Date = "abc" });
> > RoundTrip(data); // 1
> > data.Add(new ProtoBlock { Date = "def" });
> > RoundTrip(data); // 2
> > data.Add(new ProtoBlock { Date = "ghi" });
> > RoundTrip(data); // 3
> > }
> > static void RoundTrip(List<ProtoBlock> list) {
> > string path = "foo.bin";
> > if (File.Exists(path)) File.Delete(path);
> > using (Stream outfile = File.Open(path, FileMode.OpenOrCreate)) {
> > ProtoBuf.Serializer.Serialize(outfile, list);
> > }
> > List<ProtoBlock> clone;
> > using (Stream inFile = File.OpenRead(path)) {
> > clone =
> > ProtoBuf.Serializer.Deserialize<List<ProtoBlock>>(inFile)
> > ?? new List<ProtoBlock>(); // blanks become null -
> > limitation
> > }
> > Debug.Assert(!ReferenceEquals(clone, list)); // different lists
> > Debug.Assert(clone.Count == list.Count); // same count
> > for (int i = 0; i < list.Count; i++) {
> > Debug.Assert(clone[i].Date == list[i].Date); // same data
> > }
> > }
> >
> > }
> >
> > 2009/12/21 jeevankodali <jeevankod...@gmail.com>
> >
> >
> >
> >
> >
> > > Sorry for the late reply, what exactly do you want in the full
> > > example, I can add the code that I am using, do you need the file as
> > > well (and how can I send the file)?
> >
> > > Thanks
> >
> > > On Dec 19, 5:09 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
> > > > Rechecked with Net20 from the r278 build, and it worked fine. Sorry
> to be
> > > a
> > > > pain, but do you have a complete example that shows the issue?
> >
> > > > Marc
> >
> > > > 2009/12/19 Marc Gravell <marc.grav...@gmail.com>
> >
> > > > > OK; I'll re-check with the Net20 variant...
> >
> > > > > 2009/12/19 jeevankodali <jeevankod...@gmail.com>
> >
> > > > > Thanks for the reply. I used r278 and Net20 folder in that zip
> file.
> >
> > > > >> On Dec 18, 2:36 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
> > > > >> > Hmmm... that should work fine; there *was* a related bug in an
> early
> > > > >> build,
> > > > >> > but...
> >
> > > > >> > Can I check which version and framework you are using? I've
> checked
> > > on
> > > > >> r275,
> > > > >> > and it works fine "as is", for both empty lists and some
> arbitrary
> > > data
> > > > >> I
> > > > >> > made up.
> >
> > > > >> > Sorry for the delay, btw - seasonal break, etc...
> >
> > > > >> > Marc Gravell (protobuf-net)
> >
> > > > >> --
> >
> > > > >> You received this message because you are subscribed to the Google
> > > Groups
> > > > >> "Protocol Buffers" group.
> > > > >> To post to this group, send email to proto...@googlegroups.com.
> > > > >> To unsubscribe from this group, send email to
> > > > >> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> <protobuf%2bunsubscr...@googlegroups.com>
> > > <protobuf%2bunsubscr...@googlegroups.com>
> > > > >> .
> > > > >> For more options, visit this group at
> > > > >>http://groups.google.com/group/protobuf?hl=en.
> >
> > > > > --
> > > > > Regards,
> >
> > > > > Marc
> >
> > > > --
> > > > Regards,
> >
> > > > Marc- Hide quoted text -
> >
> > > > - Show quoted text -
> >
> > > --
> >
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> <protobuf%2bunsubscr...@googlegroups.com>
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
> >
> > --
> > Regards,
> >
> > Marc- Hide quoted text -
> >
> > - Show quoted text -
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
--
Regards,
Marc
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
