[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Tamás, I may not be understanding correctly, but after taking a look 
at github.com/godror/godror, and the odpi subdirectory,
I see it is including all the `.c` files on the fly 
.

A couple of reasons immediately come to mind, that make this not a 
generically valid option:

* ODPI library is all C code (see src 
) so it works 
including in Go: my dependencies are C++/Rust code, for which I write a 
small C wrapper (or for Rust just `extern "C"`). Also architecture 
dependent compilation is different in C++/C/Rust code ...
* The C++ libraries I'm including have sub-dependencies of themselves (one 
of which is llvm). It uses Bazel to organize it, and to manually move all 
the required C++ files to a directory would be years of work :) Plus would 
require me to slave to maintain things in sync. 
* The C++ libraries take hours to compile ... I don't want to impose this 
to users of my libraries.

I think the only way to work this out is distributing the pre-compiled 
C++/Rust libraries, so the Go simply refer to them (and we get the fast 
compilation times from Go). But then, how to best distribute them in an 
automatic fashion, so that users don't need to one by one figure out how to 
install them (and clean up them later if they are no longer using...) ?

Or maybe there is another convenient way I'm not seeing ?


On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:

> Can't you build (make go build for you) those libraries?
> For example, see github.com/godror/godror just includes the sources of 
> that third party library in an "odpi" subdir, and with
> ```
> /*
> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>
> #include "dpi.c"
>
> */
> import "C"
> ```
> it is compiled automatically.
>
>
> Caveat: for "go get" to download a directory, it must include a sth.go 
> file ("require.go" in the odpi/* subdirs).
> But it may work that your precompiled libs in a subdir, with a mock .go 
> file gets downloaded.
> But how will it found by your lib/app ?
>
> Tamás
>
>
> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>
>> Thanks Richard. Indeed, as you pointed out the downside is the bloating 
>> of the git repo, but it makes sense.
>>
>> But does the user have to manually clone the repository and move the `.a` 
>> file to, let's say, /usr/local/lib, or does a simple `go get` magically 
>> does everything ?
>>
>>
>> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>>
>>> It isn't a great solution, but I currently include the built library 
>>> files and necessary headers in the git repo alongside the Go code. You can 
>>> see an example here 
>>> https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
>>> include the skia library I built for use in my UI framework, unison.
>>>
>>> The main downside of this is bloating the git repo with the binary .a 
>>> and .dll files... but I've not found a better way to handle it. glfw, which 
>>> unison also depends upon, does something very similar.
>>>
>>> - Rich
>>>
>>> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>>>
 hi,

 I'm developing a couple of ML framework/libraries for Go that depend on 
 C/C++ code. Once C-libraries dependencies are installed, the CGO 
 integration work great.

 Now, for end-users that just want to use these Go libraries, having to 
 figure out how to manually build and install those C/C++/Rust dependencies 
 is a hassle -- sadly each one with a different type of build system.

 I offer pre-built `.tgz` files (for a limited set of architectures) 
 with the required `.h` and `.a/.so` files along the releases, which 
 facilitates. But it's still a hassle to install -- and no auto-uninstall 
 if 
 someone is no longer using the Go module.

 I was wondering if others have figured out how to handle this in a 
 nicer way.  Is there a recommended way to distribute prebuilt CGO 
 dependencies ?

 I like how Python wheels (`.whl` files) solve the issue, by including 
 the pre-built libraries in a sub-directory of the python library 
 installation. I was hoping there would be a similar way (maybe with a 
 separate tool) to integrate with `go.mod`. 

 Any pointers, thoughts or suggestions are very appreciated.

 Thanks!
 Jan

 ps: While searching I saw similar questions, but none that exactly 
 answered this aspect of distribution. Just in case, apologies if it's a 
 duplicate question.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nut

Re: [go-nuts] bufio.Scanner - possible bug or doc err?

[Ian Lance Taylor]

On Thu, Oct 12, 2023 at 11:42 PM 'Mark' via golang-nuts
 wrote:
>
> Yes, I can see now.
>
> Perhaps consider changing:
>
> Programs that need more control over error handling or large tokens, or must 
> run sequential scans on a reader, should use bufio.Reader instead.
>
> to:
>
> Programs that need more control over error handling or large tokens (such as 
> lines longer than MaxScanTokenSize), or must run sequential scans on a 
> reader, should use bufio.Reader instead.

Thanks, instead of that I added a link to Scanner.Buffer
(https://go.dev/cl/535216).  I hope that will help guide people in the
right direction.

Ian


> On Thursday, October 12, 2023 at 8:56:05 PM UTC+1 Ian Lance Taylor wrote:
>>
>> On Thu, Oct 12, 2023 at 10:21 AM 'Mark' via golang-nuts
>>  wrote:
>> >
>> > The docs for bufio.Scanner do say
>> > "Programs that need more control over error handling or large tokens, or 
>> > must run sequential scans on a reader, should use bufio.Reader instead"
>> > Perhaps it would be more helpful to mention what the token length limit is?
>>
>> It's MaxScanTokenSize: https://pkg.go.dev/bufio#pkg-constants . See
>> also https://pkg.go.dev/bufio#Scanner.Buffer .
>>
>> Ian
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0392f8b3-a006-4bc0-aa54-3759aa0d3b7en%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUOUHbxEk%3DQM0yyVN68JMNDeFn_yJwi4B4RrA6s6SE1Qg%40mail.gmail.com.