[go-nuts] Re: Panic in go1.8rc3 using cgo to get binary data from database
Thanks Daniel, that seems on the right track.
The driver api itself is a C library, without source code.
>From the driver api docs:
/**
* get_column() fills the supplied buffer with the value fetched for the
* specified column. *For A_BINARY and A_STRING * data types,*
* * value->buffer will point to an internal buffer associated with the
result set.*
* The content of the pointer buffer should not be relied on or altered
* as it will change when a new row is fetched or when the result set
* object is freed. Users should copy the data out of those pointers into
their
* own buffers. The length field indicates the number of valid characters
that
* buffer->buffer points to. The data returned in buffer->buffer is not
* null-terminated. This function fetches all of the returned value from the
server.
* For example, if the column contains
* a 2GB blob, this function will attempt to allocate enough memory to hold
that value.
* If this is not desired, get_data() should be used instead.
*
* returns 1 on success or 0 for failure. A failure can happen if any of the
parameters is invalid
* or if there is not enough memory to retrieve the full value from the
server.
*/
get_column( a_stmt * stmt, u32 col_index, a_data_value * value );
Here's an outline of a C implementation. This is what I want to replace
with Go:
typedef struct a_data_value
{
char * buffer; //allocated by caller except for binary & string type
size_t buffer_size; //set by caller except for binary & string type
size_t * length; //The number of valid bytes in the buffer. Must be
less than buffer_size
a_data_type type;
bool * is_null;
} a_data_value;
char * blob;
size_t size;
a_data_value value;
err = get_column( stmt_handle, column_index, )
char * buffer = (char *)emalloc( *(value.length) );
memcpy( buffer, val.buffer, *(value.length) );
size = *(val.length)
An attempt in Go, occasionally panics:
value := new(C.a_sqlany_data_value)
C.sqlany_get_column(r.stmt.ptr, col_index, value)
blob := C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length))
Another attempt, allocating value in c memory, here the call to free
occasionally panics:
value := (*C.a_data_value)(C.malloc(C.sizeof_a_data_value))
defer C.free(unsafe.Pointer(value))
C.sqlany_get_column(r.stmt.ptr, col_index, value)
blob := C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length))
Assigning the value directly appears to panic more frequently:
blob = (*[1 << 30]byte)(unsafe.Pointer(r.value.buffer))[:*value.length:*
value.length]
Note:
- null checks and get_column error check aren't shown here.
- each query returns many rows each with a single binary column value up
to around 10Mb
Any suggestions appreciated!
Thanks,
Mark
On Monday, 13 February 2017 12:51:28 UTC+13, Daniel Theophanes wrote:
>
> It is difficult to be certain, but I would put my bets on your sql
> anywhere driver may not be handling the c data lifetimes correctly, or
> might not be keeping a reference. It is difficult to tell without the
> driver code.
>
> Thanks, -Daniel
>
> On Saturday, February 11, 2017 at 7:27:11 PM UTC-8, Mark Crook wrote:
>>
>> Hello,
>>
>> I'm seeing a panic in go version go1.8rc3 linux/amd64 while trying to
>> write a database driver using cgo. The go code runs in a docker container,
>> the database in another.
>>
>> Generally it works reliably, but occasionally, during repeated retrieval
>> of binary data (photos approx 100K to 2M), the code panics.
>>
>> I haven't isolated a reproducible example, but I think it's related to
>> this code:
>>
>>
>> value := new(C.a_sqlany_data_value)
>>
>> //load the column value binary data into value.buffer
>>
>> C.sqlany_get_column(r.stmt.ptr, C.sacapi_u32(i), value)
>>
>>
>> //copy the value buffer from C to Go
>>
>> val = C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length))
>>
>>
>> I haven't seen the panic if the last line is commented out.
>>
>> Below is a stack trace. Perhaps it panics when a garbage collection is
>> triggered while waiting for C.GoBytes to return?
>>
>> Any suggestions?
>>
>> Thank you,
>> Mark
>>
>>
>>
>> runtime: unexpected return pc for runtime.sigpanic called from
>> 0xc420e1a000
>> fatal error: unknown caller pc
>>
>> runtime stack:
>> runtime.throw(0x59a90e, 0x11)
>> /go/src/runtime/panic.go:596 +0x95
>> runtime.gentraceback(0x, 0xc42002a5d8, 0x0, 0xc4200c49c0,
>> 0x0, 0x0, 0x7fff, 0x7fb675ad5cb8, 0x0, 0x0, ...)
>> /go/src/runtime/traceback.go:317 +0x13ea
>> runtime.scanstack(0xc4200c49c0, 0xc420021828)
>> /go/src/runtime/mgcmark.go:
[go-nuts] Re: Panic in go1.8rc3 using cgo to get binary data from database
Thanks Damian On Monday, 13 February 2017 04:29:59 UTC+13, Damian Gryski wrote: > > Please file an issue at https://golang.org/issues > > Damian > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Panic in go1.8rc3 using cgo to get binary data from database
Hello, I'm seeing a panic in go version go1.8rc3 linux/amd64 while trying to write a database driver using cgo. The go code runs in a docker container, the database in another. Generally it works reliably, but occasionally, during repeated retrieval of binary data (photos approx 100K to 2M), the code panics. I haven't isolated a reproducible example, but I think it's related to this code: value := new(C.a_sqlany_data_value) //load the column value binary data into value.buffer C.sqlany_get_column(r.stmt.ptr, C.sacapi_u32(i), value) //copy the value buffer from C to Go val = C.GoBytes(unsafe.Pointer(value.buffer), C.int(value.length)) I haven't seen the panic if the last line is commented out. Below is a stack trace. Perhaps it panics when a garbage collection is triggered while waiting for C.GoBytes to return? Any suggestions? Thank you, Mark runtime: unexpected return pc for runtime.sigpanic called from 0xc420e1a000 fatal error: unknown caller pc runtime stack: runtime.throw(0x59a90e, 0x11) /go/src/runtime/panic.go:596 +0x95 runtime.gentraceback(0x, 0xc42002a5d8, 0x0, 0xc4200c49c0, 0x0, 0x0, 0x7fff, 0x7fb675ad5cb8, 0x0, 0x0, ...) /go/src/runtime/traceback.go:317 +0x13ea runtime.scanstack(0xc4200c49c0, 0xc420021828) /go/src/runtime/mgcmark.go:842 +0x265 runtime.newstack(0x0) /go/src/runtime/stack.go:1064 +0x41e runtime.morestack() /go/src/runtime/asm_amd64.s:398 +0x86 goroutine 11 [GC worker (idle) (scan)]: runtime.assertE2I2(0x56c220, 0x569e20, 0x841d20, 0xc42002a6b0, 0xc4200c49e0, 0xc42002a640) /go/src/runtime/iface.go:289 +0xb2 fp=0xc42002a5e0 sp=0xc42002a5d8 runtime.preprintpanics(0xc42002a6b0) /go/src/runtime/panic.go:389 +0x9f fp=0xc42002a650 sp=0xc42002a5e0 panic(0x569e20, 0x841d20) /go/src/runtime/panic.go:528 +0x1ab fp=0xc42002a6e8 sp=0xc42002a650 runtime.panicmem() /go/src/runtime/panic.go:63 +0x5e fp=0xc42002a708 sp=0xc42002a6e8 runtime.sigpanic() /go/src/runtime/signal_unix.go:290 +0x29f fp=0xc42002a758 sp=0xc42002a708 created by runtime.gcBgMarkStartWorkers /go/src/runtime/mgc.go:1410 +0x98 goroutine 1 [chan receive]: testing.(*T).Run(0xc420072820, 0x598622, 0x8, 0x5a3718, 0xc420051d01) /go/src/testing/testing.go:698 +0x2f4 testing.runTests.func1(0xc420072820) /go/src/testing/testing.go:881 +0x67 testing.tRunner(0xc420072820, 0xc420051de0) /go/src/testing/testing.go:657 +0x96 testing.runTests(0xc42000cb60, 0x8461e0, 0x4, 0x4, 0xc420051ef8) /go/src/testing/testing.go:887 +0x2c1 testing.(*M).Run(0xc420051f20, 0xc420051f20) /go/src/testing/testing.go:822 +0xfc main.main() github.com/mdcnz/sqlanywhere/_test/_testmain.go:48 +0xf7 goroutine 17 [syscall, 6 minutes, locked to thread]: runtime.goexit() /go/src/runtime/asm_amd64.s:2197 +0x1 goroutine 231 [GC assist wait]: database/sql.convertAssign(0x5517a0, 0xc4200c20a0, 0x55a040, 0xc42000c0e0, 0x85b5c0, 0x4535c0) /go/src/database/sql/convert.go:161 +0x1a22 database/sql.(*Rows).Scan(0xc42001c6c0, 0xc420041f68, 0x1, 0x1, 0x0, 0x0) /go/src/database/sql/sql.go:2342 +0xca github.com/mdcnz/sqlanywhere.TestBlob(0xc4200728f0) /root/gome/src/github.com/mdcnz/sqlanywhere/test.go:26 +0x2a5 testing.tRunner(0xc4200728f0, 0x5a3718) /go/src/testing/testing.go:657 +0x96 created by testing.(*T).Run /go/src/testing/testing.go:697 +0x2ca goroutine 233 [chan receive]: database/sql.(*Rows).awaitDone(0xc42001c6c0, 0x84b240, 0xc42047e080) /go/src/database/sql/sql.go:2091 +0x54 created by database/sql.(*Rows).initContextClose /go/src/database/sql/sql.go:2086 +0x92 goroutine 232 [chan receive]: database/sql.(*DB).connectionOpener(0xc42008c210) /go/src/database/sql/sql.go:835 +0x4a created by database/sql.Open /go/src/database/sql/sql.go:580 +0x1c8 exit status 2 -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
