An easy "comfort" exercise: save 1000 images to a directory in png form. decode with go and with several tools such as imagemagick compare times
for format in jpeg, tiff, ... convert all images to that format; do the test above anew the result will be a comparison chart of Go existing image decoders vs X in various image formats for your images. if it is within a few percent, you can be comfortable if it is much slower, be uncomfortable and report a performance bug. if it is faster, be happy. michael On Sat, Jun 27, 2020 at 5:31 AM Robert Engels <reng...@ix.netcom.com> wrote: > Just because the bulk of the time is in the decode doesn’t mean the decode > is inefficient or can be improved upon. It might be the most expensive > stage in the process regardless of the implementation. > > On Jun 27, 2020, at 12:15 AM, Lee Armstrong <l...@pinkfroot.com> wrote: > > > Thanks Rob, > > Yes you are right these images are small. <100kb each with an example > below. And as I mentioned I have millions of these. > > I am able to max the CPUs out with a worker pool but that is when I > noticed a lot of the work was actually decoding the PNG images which made > we wonder if if was able to reuse decoders at all and even if that would > help! > > > On Sat, 27 Jun 2020 at 01:30, Rob Pike <r...@golang.org> wrote: > >> Without information about the particular images - the data set you're >> working with - it's hard to provide any accurate advice. If the images are >> small, maybe it's all overhead. If the images are of a particular form of >> PNG, maybe a fast path is missing. And so on. >> >> To get meaningful help for problems like this, the more information you >> can provide, the better. >> >> -rob >> >> >> On Fri, Jun 26, 2020 at 10:59 PM Lee Armstrong <l...@pinkfroot.com> wrote: >> >>> Thanks, I am already maxing out some servers but wondered if it could be >>> sped up. >>> >>> I will give the bimg package a go! >>> >>> On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote: >>> >>>> Just parallelize in the cloud. At a minimum parallelize locally with >>>> multiple Go routines. >>>> >>>> On Jun 26, 2020, at 7:51 AM, howar...@gmail.com wrote: >>>> >>>> >>>> >>>> I don't know if the CGo transitions for 16 million images will >>>> completely swamp the speed gains, or how much post-processing you need to >>>> do to these images, but you might try https://github.com/h2non/bimg and >>>> see if it gives you any wins. It claims 'typically 4x faster' then the Go >>>> Image package. It is a CGO interface to libvips, which is an experimental >>>> branch of libspng, which is a performance focused alternative to libpng. >>>> >>>> Howard >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golang-nuts...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com >>>> <https://groups.google.com/d/msgid/golang-nuts/9f8d3086-75f4-47e1-b49a-4dce057258cco%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts+unsubscr...@googlegroups.com. >> >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com >>> <https://groups.google.com/d/msgid/golang-nuts/d694c7d7-8438-4d7c-93ab-46cccaf09b65n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAC5LoL_yWxgfDUCs5UwStep-dOMoWovpF_RK0K_MuPNzsG6Unw%40mail.gmail.com > <https://groups.google.com/d/msgid/golang-nuts/CAC5LoL_yWxgfDUCs5UwStep-dOMoWovpF_RK0K_MuPNzsG6Unw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/D4EB5766-01C2-4025-898E-0768A302C8DF%40ix.netcom.com > <https://groups.google.com/d/msgid/golang-nuts/D4EB5766-01C2-4025-898E-0768A302C8DF%40ix.netcom.com?utm_medium=email&utm_source=footer> > . > -- *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>* -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALoEmQyKuqY2pMsOqT_7dgZLHQR_BC3r5hsXqxU6T%3DfuoOSRNQ%40mail.gmail.com.
