Thanks for the tip! Could you suggest some resources for the same regarding these design considerations if you know of any ?
On Sun, 11 May 2025, 21:48 Robert Engels, <reng...@ix.netcom.com> wrote: > Of that I’m not sure. It is based on deep knowledge of how these things > are built from the hardware to the kernel to network protocols to the > service layers. > > It broadly falls under performance tuning of which there is lots of > literature. > > The key element of how channels and go routines play into this - is that > when dealing with synchronous requests to a service (many services offer > async endpoints) you need to be able to parallelize these. Then you need to > be able to provide the data as fast as the network can accept it - most > networks today are faster than disk storage - even some SSD. So then the > disk storage array becomes the bottleneck and you need to parallelize that > (RAID etc) and the kernel access. > > Often for massive uploads you might even partition the job across machines > each with a portion of the data set. > > On May 11, 2025, at 11:04 AM, Kanak Bhatia <kanakbhati...@gmail.com> > wrote: > > > > Does we have any kind of documented material or mathematical theory type > stuff for these things or is it more like hit or try > > On Sun, 11 May 2025, 08:04 ren...@ix.netcom.com, <reng...@ix.netcom.com> > wrote: > >> The two most likely limiters in performance will be your network pipe to >> the cloud and the QPS quota offered by the service. If you are not reaching >> those limits you should increase the parallelism until you do. If your cpu >> becomes saturated first you probably need larger buffer sizes in the I/o. >> >> On Thursday, May 8, 2025 at 3:48:36 PM UTC-5 Kanak Bhatia wrote: >> >>> Does anyone have idea how to optimize consumers, producers and channels >>> using golang. I have to upload a million objects to a cloud object storage, >>> but unable to get a optimzed values for above parametres. Producers used to >>> create objects and send data through channels and receive them at >>> consumers and call api from there. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion visit >> https://groups.google.com/d/msgid/golang-nuts/6af4acac-c8c8-462c-aad4-08e1df73f694n%40googlegroups.com >> <https://groups.google.com/d/msgid/golang-nuts/6af4acac-c8c8-462c-aad4-08e1df73f694n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CABXaED4R1HLUAL7CdWHs%3DDSAZ5eiAwdpjOrPYK8f6hZDbz%2BtXA%40mail.gmail.com.
