considering it is primarily static information, maybe it should be passed in and as data during run time, instead of being embedded and compiled as code?
- sean On Wed, Jul 12, 2023, 13:18 Jason Kulatunga <darkmeth...@gmail.com> wrote: > Hi, > > I’m doing something a bit unusual with my application and I’m trying to > figure out if there’s a better way to do this that I’m missing. > > Basically my application Fasten Health > <https://github.com/fastenhealth/fasten-onprem> is designed to allow > patients to pull their medical records from healthcare institutions — of > with there are 10,000 currently supported, and 100,000s more that we'd like > to support in the US. > > We have a repo/library called fasten-sources > <https://github.com/fastenhealth/fasten-sources> which is made up of > mostly generated code (using dave/jennifer > <https://github.com/dave/jennifer>). > It has a couple of primary packages: > > > - pkg > > <https://github.com/fastenhealth/fasten-sources/blob/main/pkg/source_type.go> > which contains a list 10,000 enums, one for each support healthcare > institution. > - definitions/internal/source > > <https://github.com/fastenhealth/fasten-sources/tree/main/definitions/internal/source> > which contains a public function defining the OAuth configuration for each > healthcare institution > - clients/internal/source > > <https://github.com/fastenhealth/fasten-sources/tree/main/clients/internal/source> > which contains a public function effectively instantiating a custom > API/Http client for each healthcare institution > > > All external references to this library use a Factory pattern - > https://github.com/fastenhealth/fasten-sources/blob/main/clients/factory/factory.go > — so the public functions do not need to be public in reality. > > > At this point the build of our application (which leverages this library) > is causing OOM issues during our Docker image build > <https://github.com/fastenhealth/fasten-onprem/actions/runs/5515748969/jobs/10056394327> > on Github Actions and custom Depot.dev build nodes (64GB of memory) > > Here’s what I’m wondering: > > > 1. Could this OOM issue be caused by the number of public symbols in > the packages? As we start supporting more and more institutions, I’d expect > this number to grow by orders of magnitude, which will definitely cause > even more OOM issues > - My functions/structs all match an interface, so if I change my > public functions to private functions, would that solve the problem in a > permanent way? > 2. Is the OOM issue caused by the number of files in a package? > Instead of having 100,000s of individual files, if I had 1 massive file, > would have have any impact? > 3. Is the OOM issue caused by the number of symbols in a single > package? Should I find a way to distribute the enums, functions across > multiple packages would have have any impact? > 4. Does the `internal` path have any impact on RAM usage during > compilation? > 5. Instead of using a switch in the Factory for instantiating the > clients and definition files, is there some other compiler friendly way to > do this instead? > > > Basically I recognize that I'm doing something a bit unusual, but I cant > find a way around it. I need to support 100,000's of institutions in the > future, so bandaid fixes (or "just use a bigger build machine") won't work > for us. A fundamental refactor of the repo is completely acceptable, since > its all generated code. I just don't know which changes will actually > impact the amount of RAM used during compilation > > Thanks! > -Jason > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/f870ec97-5ac5-4211-9379-5f2c0dd97e0bn%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/f870ec97-5ac5-4211-9379-5f2c0dd97e0bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGabyPrM_G6yzh19mLHCcKAkY4cSUMPAzXhCkQdSNZ35aEdUJg%40mail.gmail.com.
