On 4/9/21 11:57 AM, Tenjin wrote:
Basically I am making a script to query an api for some json data, if
I run it synchronously it works fine and works as intended but it
needs to do this give or take 15thousand times so I thought I would
use concurrency to get the job done. It works fine for about 4
thousand queries then it stops working and gives me this error
"socket: too many open files" when I did research on the issue I
implemented everything they did I am consuming the response body and
then I am closing it once I am finished as well as letting my wait
group know I am finished to exit the go routine.
*Right now I am making a new goroutine for each request and then
closing it*. I am also using WSL2 to run the code as I came across a
stack overflow post that said I need to up my ulimit on the machine
which I cannot do and ideally do not want others to have to do.
This is most likely your problem. I can't be sure, but it sounds like
you are possibly launching all of these goroutines at once on startup.
If so, then they are likely all trying to open a network connection at
the same time. Instead, I would suggest to push your requests into a
channel and have a well defined goroutine that iterates over that
channel to service the requests. Then you can experiment with launching
N instances of your goroutine when your application starts up. This
should give you better control over the number of outstanding sockets,
and you can play with N to get good performance.
I have been stuck on this for a few days now and no one is helping me
thus far, fingers crossed this post works out well and I can get it
resolved.
--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to golang-nuts+unsubscr...@googlegroups.com
<mailto:golang-nuts+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/09e4caf4-d79c-450e-a544-2d9242167f87n%40googlegroups.com
<https://groups.google.com/d/msgid/golang-nuts/09e4caf4-d79c-450e-a544-2d9242167f87n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/185b0d53-22f9-6c6b-7cbc-739c79d1798b%40gmail.com.
