Re: [go-nuts] Re: Speed up png.Decode

2020-06-27 Thread David Riley
On Jun 27, 2020, at 8:30 AM, Robert Engels wrote: > > Just because the bulk of the time is in the decode doesn’t mean the decode is > inefficient or can be improved upon. It might be the most expensive stage in > the process regardless of the implementation. This is I think the most

Re: [go-nuts] Re: Speed up png.Decode

2020-06-27 Thread Michael Jones
An easy "comfort" exercise: save 1000 images to a directory in png form. decode with go and with several tools such as imagemagick compare times for format in jpeg, tiff, ... convert all images to that format; do the test above anew the result will be a comparison chart of Go existing image

Re: [go-nuts] Re: Speed up png.Decode

2020-06-27 Thread Robert Engels
Just because the bulk of the time is in the decode doesn’t mean the decode is inefficient or can be improved upon. It might be the most expensive stage in the process regardless of the implementation. > On Jun 27, 2020, at 12:15 AM, Lee Armstrong wrote: > >  > Thanks Rob, > > Yes you are

Re: [go-nuts] Re: Speed up png.Decode

2020-06-26 Thread Lee Armstrong
Thanks Rob, Yes you are right these images are small. <100kb each with an example below. And as I mentioned I have millions of these. I am able to max the CPUs out with a worker pool but that is when I noticed a lot of the work was actually decoding the PNG images which made we wonder if if was

Re: [go-nuts] Re: Speed up png.Decode

2020-06-26 Thread Rob Pike
Without information about the particular images - the data set you're working with - it's hard to provide any accurate advice. If the images are small, maybe it's all overhead. If the images are of a particular form of PNG, maybe a fast path is missing. And so on. To get meaningful help for

Re: [go-nuts] Re: Speed up png.Decode

2020-06-26 Thread Lee Armstrong
Thanks, I am already maxing out some servers but wondered if it could be sped up. I will give the bimg package a go! On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote: > Just parallelize in the cloud. At a minimum parallelize locally with > multiple Go routines. > > On

Re: [go-nuts] Re: Speed up png.Decode

2020-06-26 Thread Robert Engels
Just parallelize in the cloud. At a minimum parallelize locally with multiple Go routines. > On Jun 26, 2020, at 7:51 AM, howardcs...@gmail.com wrote: > >  > I don't know if the CGo transitions for 16 million images will completely > swamp the speed gains, or how much post-processing you

[go-nuts] Re: Speed up png.Decode

2020-06-26 Thread howardcshaw
I don't know if the CGo transitions for 16 million images will completely swamp the speed gains, or how much post-processing you need to do to these images, but you might try https://github.com/h2non/bimg and see if it gives you any wins. It claims 'typically 4x faster' then the Go Image