[protobuf] segfault when deserializing under high concurrency
So part of this is a mono bug, segfaults shouldn't be happening in managed code period. I'm embedding mono 3 in a java app and consistently get the following when deserializing a simple protobuf message. Stacktrace: at unknown 0x at (wrapper managed-to-native) System.Buffer.BlockCopyInternal (System.Array,int,System.Array,int,int) 0x at System.Buffer.BlockCopy (System.Array,int,System.Array,int,int) 0x0006b at System.IO.MemoryStream.Read (byte[],int,int) 0x000ff at ProtoBuf.ProtoReader.Ensure (int,bool) 0x00237 at ProtoBuf.ProtoReader.TryReadUInt32VariantWithoutMoving (bool,uint) 0x00043 at ProtoBuf.ProtoReader.TryReadUInt32Variant (uint) 0x0001f at ProtoBuf.ProtoReader.ReadFieldHeader () 0x00057 at (wrapper dynamic-method) com.game_machine.entity_system.generated.Entity.proto_2 (object,ProtoBuf.ProtoReader) 0x02802 at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) 0x0003f at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) 0x00150 at ProtoBuf.Meta.TypeModel.DeserializeCore (ProtoBuf.ProtoReader,System.Type,object,bool) 0x00064 at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type,ProtoBuf.SerializationContext) 0x0009b at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type) 0x0001f at ProtoBuf.Serializer.DeserializeT (System.IO.Stream) 0x00043 at GameMachine.Actor.ByteArrayToEntity (byte[]) 0x00047 at GameMachine.TestActor.OnReceive (byte[]) 0x0006b at (wrapper runtime-invoke) Module.runtime_invoke_void__this___object (object,intptr,intptr,intptr) 0x This goes away if I just run in a single thread. It also takes an average of 10,000 iterations or so to trigger this error. I also tried wrapping all deserialization calls in a mutex but that had no effect. The threading model is such that none of the objects I am deserializing are accessed concurrently by different threads. Is protobuf-net completely reentrant, or does it try to reuse objects anywhere? This could very well all be a mono bug, but thought I would check here first to see if someone had any ideas. Chris -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] segfault when deserializing under high concurrency
Firstly: exactly what version is this? There was a bug in 663 relating to threading (and which only exhibited after extended usage) that was fixed in something like 668. If you are using something = 663 and 668 then please update and retry. Marc (protobuf-net) On 15 Oct 2013 16:22, ch...@ochsnet.com wrote: So part of this is a mono bug, segfaults shouldn't be happening in managed code period. I'm embedding mono 3 in a java app and consistently get the following when deserializing a simple protobuf message. Stacktrace: at unknown 0x at (wrapper managed-to-native) System.Buffer.BlockCopyInternal (System.Array,int,System.Array,int,int) 0x at System.Buffer.BlockCopy (System.Array,int,System.Array,int,int) 0x0006b at System.IO.MemoryStream.Read (byte[],int,int) 0x000ff at ProtoBuf.ProtoReader.Ensure (int,bool) 0x00237 at ProtoBuf.ProtoReader.TryReadUInt32VariantWithoutMoving (bool,uint) 0x00043 at ProtoBuf.ProtoReader.TryReadUInt32Variant (uint) 0x0001f at ProtoBuf.ProtoReader.ReadFieldHeader () 0x00057 at (wrapper dynamic-method) com.game_machine.entity_system.generated.Entity.proto_2 (object,ProtoBuf.ProtoReader) 0x02802 at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) 0x0003f at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) 0x00150 at ProtoBuf.Meta.TypeModel.DeserializeCore (ProtoBuf.ProtoReader,System.Type,object,bool) 0x00064 at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type,ProtoBuf.SerializationContext) 0x0009b at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type) 0x0001f at ProtoBuf.Serializer.DeserializeT (System.IO.Stream) 0x00043 at GameMachine.Actor.ByteArrayToEntity (byte[]) 0x00047 at GameMachine.TestActor.OnReceive (byte[]) 0x0006b at (wrapper runtime-invoke) Module.runtime_invoke_void__this___object (object,intptr,intptr,intptr) 0x This goes away if I just run in a single thread. It also takes an average of 10,000 iterations or so to trigger this error. I also tried wrapping all deserialization calls in a mutex but that had no effect. The threading model is such that none of the objects I am deserializing are accessed concurrently by different threads. Is protobuf-net completely reentrant, or does it try to reuse objects anywhere? This could very well all be a mono bug, but thought I would check here first to see if someone had any ideas. Chris -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] segfault when deserializing under high concurrency
I was using 668, but this is a mono bug with reading streams I confirmed it with just reading from the stream myself. Rather annoying, I updated to the latest mono 3 master because of a threading bug that was fixed 2 weeks ago, now this:) Chris On Tuesday, October 15, 2013 10:45:46 AM UTC-7, Marc Gravell wrote: Firstly: exactly what version is this? There was a bug in 663 relating to threading (and which only exhibited after extended usage) that was fixed in something like 668. If you are using something = 663 and 668 then please update and retry. Marc (protobuf-net) On 15 Oct 2013 16:22, ch...@ochsnet.com javascript: wrote: So part of this is a mono bug, segfaults shouldn't be happening in managed code period. I'm embedding mono 3 in a java app and consistently get the following when deserializing a simple protobuf message. Stacktrace: at unknown 0x at (wrapper managed-to-native) System.Buffer.BlockCopyInternal (System.Array,int,System.Array,int,int) 0x at System.Buffer.BlockCopy (System.Array,int,System.Array,int,int) 0x0006b at System.IO.MemoryStream.Read (byte[],int,int) 0x000ff at ProtoBuf.ProtoReader.Ensure (int,bool) 0x00237 at ProtoBuf.ProtoReader.TryReadUInt32VariantWithoutMoving (bool,uint) 0x00043 at ProtoBuf.ProtoReader.TryReadUInt32Variant (uint) 0x0001f at ProtoBuf.ProtoReader.ReadFieldHeader () 0x00057 at (wrapper dynamic-method) com.game_machine.entity_system.generated.Entity.proto_2 (object,ProtoBuf.ProtoReader) 0x02802 at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read (object,ProtoBuf.ProtoReader) 0x0003f at ProtoBuf.Meta.RuntimeTypeModel.Deserialize (int,object,ProtoBuf.ProtoReader) 0x00150 at ProtoBuf.Meta.TypeModel.DeserializeCore (ProtoBuf.ProtoReader,System.Type,object,bool) 0x00064 at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type,ProtoBuf.SerializationContext) 0x0009b at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream,object,System.Type) 0x0001f at ProtoBuf.Serializer.DeserializeT (System.IO.Stream) 0x00043 at GameMachine.Actor.ByteArrayToEntity (byte[]) 0x00047 at GameMachine.TestActor.OnReceive (byte[]) 0x0006b at (wrapper runtime-invoke) Module.runtime_invoke_void__this___object (object,intptr,intptr,intptr) 0x This goes away if I just run in a single thread. It also takes an average of 10,000 iterations or so to trigger this error. I also tried wrapping all deserialization calls in a mutex but that had no effect. The threading model is such that none of the objects I am deserializing are accessed concurrently by different threads. Is protobuf-net completely reentrant, or does it try to reuse objects anywhere? This could very well all be a mono bug, but thought I would check here first to see if someone had any ideas. Chris -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com javascript:. To post to this group, send email to prot...@googlegroups.comjavascript: . Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] store large data into files using protobuf
Hi All, i wanna store 1 billion unique numbers into files using protocol buffer but the default file size limit is set 64MB how can i store 1 billion numbers into file. Thanks -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
