Re: [go-nuts] Re: Speed up png.Decode

[David Riley]

On Jun 27, 2020, at 8:30 AM, Robert Engels  wrote:
> 
> Just because the bulk of the time is in the decode doesn’t mean the decode is 
> inefficient or can be improved upon. It might be the most expensive stage in 
> the process regardless of the implementation. 

This is I think the most important point; image decoding is an expensive 
operation, and it makes sense that the bulk of your time decoding is spent on 
it.  The question is how it stacks up to other decoders; if you're able to 
write a quick C proof-of-concept to test how fast libpng does it (or libspng) 
and run the same benchmarks, you'll get your answer there.

Depending on how regular your images are (size-wise, especially), you may be 
able to get some additional mileage out of pooling your receiving buffers and 
reusing them after your operations are complete. There doesn't appear to be a 
handle for that in the current image/png library like there is for encoding, 
though; you'd have to do some surgery on decoder.readImagePass to do something 
like that.


- Dave

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5F296DCC-1521-4D04-9D93-9BCEB6B5B537%40gmail.com.

Re: [go-nuts] Re: Speed up png.Decode

[Michael Jones]

An easy "comfort" exercise:

save 1000 images to a directory in png form.
decode with go and with several tools such as imagemagick
compare times

for format in jpeg, tiff, ...
convert all images to that format; do the test above anew

the result will be a comparison chart of Go existing image decoders vs X in
various image formats for your images.

if it is within a few percent, you can be comfortable
if it is much slower, be uncomfortable and report a performance bug.
if it is faster, be happy.

michael

On Sat, Jun 27, 2020 at 5:31 AM Robert Engels  wrote:

> Just because the bulk of the time is in the decode doesn’t mean the decode
> is inefficient or can be improved upon. It might be the most expensive
> stage in the process regardless of the implementation.
>
> On Jun 27, 2020, at 12:15 AM, Lee Armstrong  wrote:
>
> 
> Thanks Rob,
>
> Yes you are right these images are small. <100kb each with an example
> below. And as I mentioned I have millions of these.
>
> I am able to max the CPUs out with a worker pool but that is when I
> noticed a lot of the work was actually decoding the PNG images which made
> we wonder if if was able to reuse decoders at all and even if that would
> help!
>
>
> On Sat, 27 Jun 2020 at 01:30, Rob Pike  wrote:
>
>> Without information about the particular images - the data set you're
>> working with - it's hard to provide any accurate advice. If the images are
>> small, maybe it's all overhead. If the images are of a particular form of
>> PNG, maybe a fast path is missing. And so on.
>>
>> To get meaningful help for problems like this, the more information you
>> can provide, the better.
>>
>> -rob
>>
>>
>> On Fri, Jun 26, 2020 at 10:59 PM Lee Armstrong  wrote:
>>
>>> Thanks, I am already maxing out some servers but wondered if it could be
>>> sped up.
>>>
>>> I will give the bimg package a go!
>>>
>>> On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote:
>>>
 Just parallelize in the cloud. At a minimum parallelize locally with
 multiple Go routines.

 On Jun 26, 2020, at 7:51 AM, howar...@gmail.com wrote:

 

 I don't know if the CGo transitions for 16 million images will
 completely swamp the speed gains, or how much post-processing you need to
 do to these images, but you might try https://github.com/h2non/bimg and
 see if it gives you any wins. It claims 'typically 4x faster' then the Go
 Image package. It is a CGO interface to libvips, which is an experimental
 branch of libspng, which is a performance focused alternative to libpng.

 Howard

 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to golang-nuts...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com
 
 .

 --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>
>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAC5LoL_yWxgfDUCs5UwStep-dOMoWovpF_RK0K_MuPNzsG6Unw%40mail.gmail.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/D4EB5766-01C2-4025-898E-0768A302C8DF%40ix.netcom.com
> 
> .
>


-- 

*Michael T. jonesmichael.jo...@gmail.com *

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails fr

Re: [go-nuts] Re: Speed up png.Decode

[Robert Engels]

Just because the bulk of the time is in the decode doesn’t mean the decode is 
inefficient or can be improved upon. It might be the most expensive stage in 
the process regardless of the implementation. 

> On Jun 27, 2020, at 12:15 AM, Lee Armstrong  wrote:
> 
> 
> Thanks Rob,
> 
> Yes you are right these images are small. <100kb each with an example below. 
> And as I mentioned I have millions of these. 
> 
> I am able to max the CPUs out with a worker pool but that is when I noticed a 
> lot of the work was actually decoding the PNG images which made we wonder if 
> if was able to reuse decoders at all and even if that would help!
> 
> 
> 
>> On Sat, 27 Jun 2020 at 01:30, Rob Pike  wrote:
>> Without information about the particular images - the data set you're 
>> working with - it's hard to provide any accurate advice. If the images are 
>> small, maybe it's all overhead. If the images are of a particular form of 
>> PNG, maybe a fast path is missing. And so on.
>> 
>> To get meaningful help for problems like this, the more information you can 
>> provide, the better.
>> 
>> -rob
>> 
>> 
 On Fri, Jun 26, 2020 at 10:59 PM Lee Armstrong  wrote:
 Thanks, I am already maxing out some servers but wondered if it could be 
 sped up.
 
 I will give the bimg package a go!
 
> On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote:
> Just parallelize in the cloud. At a minimum parallelize locally with 
> multiple Go routines. 
> 
>>> On Jun 26, 2020, at 7:51 AM, howar...@gmail.com wrote:
>>> 
>> 
> 
>> I don't know if the CGo transitions for 16 million images will 
>> completely swamp the speed gains, or how much post-processing you need 
>> to do to these images, but you might try https://github.com/h2non/bimg 
>> and see if it gives you any wins. It claims 'typically 4x faster' then 
>> the Go Image package. It is a CGO interface to libvips, which is an 
>> experimental branch of libspng, which is a performance focused 
>> alternative to libpng.
>> 
>> Howard
> 
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com.
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to golang-nuts+unsubscr...@googlegroups.com.
>>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAC5LoL_yWxgfDUCs5UwStep-dOMoWovpF_RK0K_MuPNzsG6Unw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/D4EB5766-01C2-4025-898E-0768A302C8DF%40ix.netcom.com.

Re: [go-nuts] Re: Speed up png.Decode

[Lee Armstrong]

Thanks Rob,

Yes you are right these images are small. <100kb each with an example
below. And as I mentioned I have millions of these.

I am able to max the CPUs out with a worker pool but that is when I noticed
a lot of the work was actually decoding the PNG images which made we wonder
if if was able to reuse decoders at all and even if that would help!


On Sat, 27 Jun 2020 at 01:30, Rob Pike  wrote:

> Without information about the particular images - the data set you're
> working with - it's hard to provide any accurate advice. If the images are
> small, maybe it's all overhead. If the images are of a particular form of
> PNG, maybe a fast path is missing. And so on.
>
> To get meaningful help for problems like this, the more information you
> can provide, the better.
>
> -rob
>
>
> On Fri, Jun 26, 2020 at 10:59 PM Lee Armstrong  wrote:
>
>> Thanks, I am already maxing out some servers but wondered if it could be
>> sped up.
>>
>> I will give the bimg package a go!
>>
>> On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote:
>>
>>> Just parallelize in the cloud. At a minimum parallelize locally with
>>> multiple Go routines.
>>>
>>> On Jun 26, 2020, at 7:51 AM, howar...@gmail.com wrote:
>>>
>>> 
>>>
>>> I don't know if the CGo transitions for 16 million images will
>>> completely swamp the speed gains, or how much post-processing you need to
>>> do to these images, but you might try https://github.com/h2non/bimg and
>>> see if it gives you any wins. It claims 'typically 4x faster' then the Go
>>> Image package. It is a CGO interface to libvips, which is an experimental
>>> branch of libspng, which is a performance focused alternative to libpng.
>>>
>>> Howard
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com
>>> 
>>> .
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>
>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAC5LoL_yWxgfDUCs5UwStep-dOMoWovpF_RK0K_MuPNzsG6Unw%40mail.gmail.com.

Re: [go-nuts] Re: Speed up png.Decode

[Rob Pike]

Without information about the particular images - the data set you're
working with - it's hard to provide any accurate advice. If the images are
small, maybe it's all overhead. If the images are of a particular form of
PNG, maybe a fast path is missing. And so on.

To get meaningful help for problems like this, the more information you can
provide, the better.

-rob


On Fri, Jun 26, 2020 at 10:59 PM Lee Armstrong  wrote:

> Thanks, I am already maxing out some servers but wondered if it could be
> sped up.
>
> I will give the bimg package a go!
>
> On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote:
>
>> Just parallelize in the cloud. At a minimum parallelize locally with
>> multiple Go routines.
>>
>> On Jun 26, 2020, at 7:51 AM, howar...@gmail.com wrote:
>>
>> 
>>
>> I don't know if the CGo transitions for 16 million images will completely
>> swamp the speed gains, or how much post-processing you need to do to these
>> images, but you might try https://github.com/h2non/bimg and see if it
>> gives you any wins. It claims 'typically 4x faster' then the Go Image
>> package. It is a CGO interface to libvips, which is an experimental branch
>> of libspng, which is a performance focused alternative to libpng.
>>
>> Howard
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com
>> 
>> .
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOXNBZTc6nOw3DimkEjMRpvy_HO8K%3D9xqPhFCt2oAYhr%3D0kZTQ%40mail.gmail.com.

Re: [go-nuts] Re: Speed up png.Decode

[Lee Armstrong]

Thanks, I am already maxing out some servers but wondered if it could be 
sped up.

I will give the bimg package a go!

On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote:

> Just parallelize in the cloud. At a minimum parallelize locally with 
> multiple Go routines. 
>
> On Jun 26, 2020, at 7:51 AM, howar...@gmail.com wrote:
>
> 
>
> I don't know if the CGo transitions for 16 million images will completely 
> swamp the speed gains, or how much post-processing you need to do to these 
> images, but you might try https://github.com/h2non/bimg and see if it 
> gives you any wins. It claims 'typically 4x faster' then the Go Image 
> package. It is a CGO interface to libvips, which is an experimental branch 
> of libspng, which is a performance focused alternative to libpng.
>
> Howard
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com
>  
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com.

Re: [go-nuts] Re: Speed up png.Decode

[Robert Engels]

Just parallelize in the cloud. At a minimum parallelize locally with multiple 
Go routines. 

> On Jun 26, 2020, at 7:51 AM, howardcs...@gmail.com wrote:
> 
> 
> I don't know if the CGo transitions for 16 million images will completely 
> swamp the speed gains, or how much post-processing you need to do to these 
> images, but you might try https://github.com/h2non/bimg and see if it gives 
> you any wins. It claims 'typically 4x faster' then the Go Image package. It 
> is a CGO interface to libvips, which is an experimental branch of libspng, 
> which is a performance focused alternative to libpng.
> 
> Howard
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4602D226-D923-4F93-A256-429D8657F5FE%40ix.netcom.com.