You won't be able to parse the data as a LogFile with the format you write,
but if you are ok with manually parsing each delimited message, that
approach will be fine.

There are probably other folks who are more familiar with the Java
internals and how to optimize for builder reuse etc.

On Wed, Apr 11, 2012 at 2:06 PM, Shrijeet Paliwal <
shrij...@rocketfuelinc.com> wrote:

> Than you Jason. Yes I profiled it and found > 50% of execution time
> was going into invokation of this method :
>
> public static ByteString copyFromUtf8(final String text)
>
> Digging further it appeared this was called again and again while
> LogFile was trying to find the size of message.
> What I decide to do was , instead of calling logfile.add(record) and
> then finally calling a single writeTo(), I instead
> called 'recordB.build().writeDelimitedTo(output)' in each iteration of
> loop. Execution time dropped by 50%.
>
> What do you think Jason? Am I making things up or this makes sense?
>
> On Apr 11, 1:37 pm, Jason Hsueh <jas...@google.com> wrote:
> > Have you profiled this program? I would be surprised if protobuf code
> > consumes a significant amount of time: all you are doing is setting the
> > field values.
> >
> > If you want to minimize memory usage, you can probably flush your LogFile
> > periodically to the stream: as long as you don't write anything else to
> the
> > stream, the concatenated data will be properly parsed into a single
> message
> > with all the LogRecords.
> >
> > On Tue, Apr 10, 2012 at 5:51 PM, Shrijeet Paliwal <
> >
> >
> >
> >
> >
> >
> >
> > shrij...@rocketfuelinc.com> wrote:
> > > Can one of more experienced members comment on this attempt
> > >https://gist.github.com/2355991
> > > It took 25 seconds to convert a 261M file using this. The heap usage
> > > was > 512M
> > > I am hoping for tips to improve performance.
> >
> > > On Apr 6, 3:14 pm, Shrijeet Paliwal <shrij...@rocketfuelinc.com>
> > > wrote:
> > > > Could some one me direct on how to parse and map a delimited string
> to
> > > > a protobuf Message, example a tab delimited string with 100 fields
> > > > needs to be mapped to respective protobuf defined message. I am
> trying
> > > > to convert a text log file to a protobuf serialized file.
> >
> > > > -Shrijeet
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to protobuf@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to