paleolimbot commented on PR #2219:
URL: https://github.com/apache/arrow-adbc/pull/2219#issuecomment-2445832133
> It seems Redshift is juuust different enough that it's not quite Postgres
anymore but not different enough to warrant a separate codebase.
I agree that it's on the knife edge!
> Is there any way to tell from libpq that we're dealing with Redshift and
just automatically disable COPY and change the type query?
It looks like `SELECT version()` does the trick here, with some parsing. I
implemented this and pushed it...I'm not sure if it's too much of a hack? (I
see there's some failing tests, so maybe)
> Would it make sense to try and do a text based COPY instead of binary? Or
does Redshift disable that altogether?
If we could to text we'd need a completely different parser (and in a funny
way, even the "use copy = false" branch is using the binary COPY format, it's
just accessing it through the `PGresult` instead of pulling the COPY using
`PQgetcopy()`.
> I'm believe tools like AWS SDK for pandas use a COPY from Parquet to
achieve high throughput to Redshift, so there might be some precedent to still
go that route
We could probably exploit that if we used a Go or Rust based driver!
Quick demo:
``` r
library(adbcdrivermanager)
db <- adbc_database_init(
adbcpostgresql::adbcpostgresql(),
uri = Sys.getenv("ADBC_REDSHIFT_TEST_URI")
)
con <- db |>
adbc_connection_init()
con |>
read_adbc("SELECT 'foofy'") |>
tibble::as_tibble()
#> # A tibble: 1 × 1
#> `?column?`
#> <chr>
#> 1 foofy
con |>
adbc_connection_get_info() |>
tibble::as_tibble()
#> # A tibble: 6 × 2
#> info_name info_value$string_value $bool_value $int64_value
$int32_bitmask
#> <dbl> <chr> <lgl> <dbl>
<int>
#> 1 0 Redshift NA NA
NA
#> 2 1 1.0.77467 NA NA
NA
#> 3 100 ADBC PostgreSQL Driver NA NA
NA
#> 4 101 (unknown) NA NA
NA
#> 5 102 0.6.0 NA NA
NA
#> 6 103 <NA> NA 1001000
NA
#> # ℹ 2 more variables: info_value$string_list <list<chr>>,
#> # $int32_to_int32_list_map <list<df[,2]>>
```
<sup>Created on 2024-10-29 with [reprex
v2.1.1](https://reprex.tidyverse.org)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
