This is an automated email from the ASF dual-hosted git repository.
klesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/devlake.git
The following commit(s) were added to refs/heads/main by this push:
new 0417eee3b refactor(deps): drop lib/pq and bump pgx/v5 to v5.10.0
(#9031)
0417eee3b is described below
commit 0417eee3bb96c094bed9e2125ade19f2f91a8145
Author: Dirk <[email protected]>
AuthorDate: Sun Aug 16 10:03:27 2026 +0200
refactor(deps): drop lib/pq and bump pgx/v5 to v5.10.0 (#9031)
* refactor(deps): drop lib/pq and bump pgx/v5 to v5.10.0
PostgreSQL support already runs entirely on pgx: `core/runner/db.go` opens
PostgreSQL connections through `gorm.io/driver/postgres`, which uses
`github.com/jackc/pgx/v5` under the hood. There is no `sql.Open("postgres",
...)`
and no blank import of `lib/pq` anywhere in the tree.
The single remaining functional use of `lib/pq` was in the StarRocks plugin,
where `pq.Array` was used to scan PostgreSQL array columns. It is replaced
by
`pgtype.Map.SQLScanner`, the pgx equivalent that implements `sql.Scanner`
and
therefore works on the `database/sql` code path used by gorm. The
`pgtype.Map`
is created once per query, outside the row loop, because building it
registers
the full default type catalog.
Shipping a second, effectively unused PostgreSQL driver also meant shipping
`pgx/v5 v5.6.0`, which is affected by two CRITICAL advisories
(GHSA-9jj7-4m8r-rfcm, GHSA-xgrm-4fwx-7qm8) plus a low-severity one
(GHSA-j88v-2chj-qfwx), all fixed in v5.9.2. Since pgx becomes a direct
dependency here, it is bumped to v5.10.0.
`gorm.io/driver/postgres` and `gorm.io/gorm` are deliberately left
untouched:
`driver/postgres v1.5.2` only declares a minimum pgx version, so raising pgx
does not require an ORM bump.
Notes for reviewers:
- `pgx/v5` moves from the indirect block to a direct requirement.
- Array literals parse identically to the previous implementation, including
empty arrays, empty elements, quoted separators and UTF-8 content. This is
covered by a new unit test; the array scan path had no test coverage
before.
- `NULL` elements inside arrays were already rejected by `pq.Array` and are
still rejected (only the error message differs) — this is not a regression
and not a silent fix.
Signed-off-by: DoDiODev <[email protected]>
* ci: bootstrap Go 1.26 in Go workflows
Signed-off-by: DoDiODev <[email protected]>
* fix(ci): mark the workspace as a git safe.directory in the lint job
The `generate mock` step of the `golangci-lint` workflow fails with
ERR encountered error when loading package
error="-: error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping."
make: *** [Makefile:80: mock] Error 1
The job runs inside `mericodev/lake-builder:latest` as root, while the
checkout is owned by the runner user, so git refuses to operate on the
repository ("dubious ownership") and Go's VCS stamping aborts while
mockery loads the packages.
`test.yml` and `test-e2e.yml` already run
`git config --global --add safe.directory $(pwd)` right after the
checkout for exactly this reason, which is why the unit-test and e2e
jobs pass while lint does not. Add the same step to the lint job.
---------
Signed-off-by: DoDiODev <[email protected]>
---
.github/workflows/golangci-lint.yml | 4 ++
.github/workflows/test-e2e.yml | 4 ++
.github/workflows/test.yml | 2 +
backend/go.mod | 23 ++++---
backend/go.sum | 50 +++++++-------
backend/plugins/starrocks/tasks/tasks.go | 7 +-
backend/plugins/starrocks/tasks/tasks_test.go | 68 +++++++++++++++++++
backend/scripts/install-go.sh | 94 +++++++++++++++++++++++++++
8 files changed, 212 insertions(+), 40 deletions(-)
diff --git a/.github/workflows/golangci-lint.yml
b/.github/workflows/golangci-lint.yml
index d625667ce..454d0f244 100644
--- a/.github/workflows/golangci-lint.yml
+++ b/.github/workflows/golangci-lint.yml
@@ -30,8 +30,12 @@ jobs:
container: mericodev/lake-builder:latest
steps:
- uses: actions/checkout@v6
+ - name: Set git config
+ run: git config --global --add safe.directory $(pwd)
- name: Install libgit2 1.5 for git2go v34
run: backend/scripts/install-libgit2.sh
+ - name: Install Go 1.26.2
+ run: backend/scripts/install-go.sh
- name: Install mockery 3.7.2
run: backend/scripts/install-mockery.sh
- name: Cache golangci-lint
diff --git a/.github/workflows/test-e2e.yml b/.github/workflows/test-e2e.yml
index c42b77530..6e17826ad 100644
--- a/.github/workflows/test-e2e.yml
+++ b/.github/workflows/test-e2e.yml
@@ -52,6 +52,8 @@ jobs:
- run: git config --global --add safe.directory $(pwd)
- name: Install libgit2 1.5 for git2go v34
run: backend/scripts/install-libgit2.sh
+ - name: Install Go 1.26.2
+ run: backend/scripts/install-go.sh
- name: Build Python
run: |
cd backend
@@ -94,6 +96,8 @@ jobs:
- run: git config --global --add safe.directory $(pwd)
- name: Install libgit2 1.5 for git2go v34
run: backend/scripts/install-libgit2.sh
+ - name: Install Go 1.26.2
+ run: backend/scripts/install-go.sh
- name: Build Python
run: |
cd backend
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index ae9d080e6..6e3f88c3a 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -49,6 +49,8 @@ jobs:
run: git config --global --add safe.directory $(pwd)
- name: Install libgit2 1.5 for git2go v34
run: backend/scripts/install-libgit2.sh
+ - name: Install Go 1.26.2
+ run: backend/scripts/install-go.sh
- name: Install mockery 3.7.2
run: backend/scripts/install-mockery.sh
- name: Build Python
diff --git a/backend/go.mod b/backend/go.mod
index 2db88572d..6e6814707 100644
--- a/backend/go.mod
+++ b/backend/go.mod
@@ -12,8 +12,7 @@ require (
github.com/go-playground/validator/v10 v10.19.0
github.com/gocarina/gocsv v0.0.0-20220707092902-b9da1f06c77e
github.com/google/uuid v1.3.0
- github.com/jackc/pgx/v5 v5.6.0 // indirect
- github.com/lib/pq v1.10.2
+ github.com/jackc/pgx/v5 v5.10.0
github.com/libgit2/git2go/v34 v34.0.0
github.com/magiconair/properties v1.8.5
github.com/mitchellh/mapstructure v1.5.0
@@ -24,16 +23,16 @@ require (
github.com/spf13/cast v1.5.1
github.com/spf13/cobra v1.5.0
github.com/spf13/viper v1.8.1
- github.com/stretchr/testify v1.9.0
+ github.com/stretchr/testify v1.11.1
github.com/swaggo/files v1.0.1
github.com/swaggo/gin-swagger v1.6.0
github.com/swaggo/swag v1.16.1
github.com/tidwall/gjson v1.14.3
github.com/viant/afs v1.16.0
- golang.org/x/crypto v0.26.0 // indirect
+ golang.org/x/crypto v0.41.0 // indirect
golang.org/x/exp v0.0.0-20221028150844-83b7d23a625f
golang.org/x/oauth2 v0.13.0
- golang.org/x/sync v0.8.0
+ golang.org/x/sync v0.17.0
gorm.io/datatypes v1.0.1
gorm.io/driver/mysql v1.5.1
gorm.io/driver/postgres v1.5.2
@@ -47,7 +46,7 @@ require (
github.com/golang-jwt/jwt/v5 v5.0.0-rc.1
github.com/merico-ai/graphql v0.0.0-20260206020408-b7fd267bcfac
github.com/rogpeppe/go-internal v1.11.0
- golang.org/x/mod v0.17.0
+ golang.org/x/mod v0.27.0
)
require (
@@ -85,8 +84,8 @@ require (
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
- github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a //
indirect
- github.com/jackc/puddle/v2 v2.2.1 // indirect
+ github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 //
indirect
+ github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 //
indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
@@ -119,10 +118,10 @@ require (
github.com/ugorji/go/codec v1.2.11 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
golang.org/x/arch v0.3.0 // indirect
- golang.org/x/net v0.25.0 // indirect
- golang.org/x/sys v0.23.0 // indirect
- golang.org/x/text v0.17.0 // indirect
- golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
+ golang.org/x/net v0.43.0 // indirect
+ golang.org/x/sys v0.35.0 // indirect
+ golang.org/x/text v0.29.0 // indirect
+ golang.org/x/tools v0.36.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
diff --git a/backend/go.sum b/backend/go.sum
index 963625ce8..71fc1e9b9 100644
--- a/backend/go.sum
+++ b/backend/go.sum
@@ -307,8 +307,8 @@ github.com/jackc/pgproto3/v2 v2.0.1/go.mod
h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwX
github.com/jackc/pgproto3/v2 v2.0.6/go.mod
h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgservicefile v0.0.0-20200307190119-3430c5407db8/go.mod
h1:vsD4gTJCa9TptPL8sPkXrLZ+hDuNrZCnj29CQpr4X1E=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b/go.mod
h1:vsD4gTJCa9TptPL8sPkXrLZ+hDuNrZCnj29CQpr4X1E=
-github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a
h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk=
-github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod
h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761
h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod
h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgtype v0.0.0-20190421001408-4ed0de4755e0/go.mod
h1:hdSHsc1V01CGwFsrv11mJRHWJ6aifDLfdV3aVjFF0zg=
github.com/jackc/pgtype v0.0.0-20190824184912-ab885b375b90/go.mod
h1:KcahbBH1nCMSo2DXpzsoWOAfFkdEtEJpPbVLq8eE+mc=
github.com/jackc/pgtype v0.0.0-20190828014616-a8802b16cc59/go.mod
h1:MWlu30kVJrUS8lot6TQqcg7mtthZ9T0EoIBFiJcmcyw=
@@ -323,15 +323,15 @@ github.com/jackc/pgx/v4 v4.5.0/go.mod
h1:EpAKPLdnTorwmPUUsqrPxy5fphV18j9q3wrfRXg
github.com/jackc/pgx/v4 v4.6.1-0.20200510190926-94ba730bb1e9/go.mod
h1:t3/cdRQl6fOLDxqtlyhe9UWgfIi9R8+8v8GKV5TRA/o=
github.com/jackc/pgx/v4 v4.6.1-0.20200606145419-4e5062306904/go.mod
h1:ZDaNWkt9sW1JMiNn0kdYBaLelIhw7Pg4qd+Vk6tw7Hg=
github.com/jackc/pgx/v4 v4.10.1/go.mod
h1:QlrWebbs3kqEZPHCTGyxecvzG6tvIsYu+A5b1raylkA=
-github.com/jackc/pgx/v5 v5.6.0 h1:SWJzexBzPL5jb0GEsrPMLIsi/3jOo7RHlzTjcAeDrPY=
-github.com/jackc/pgx/v5 v5.6.0/go.mod
h1:DNZ/vlrUnhWCoFGxHAG8U2ljioxukquj7utPDgtQdTw=
+github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
+github.com/jackc/pgx/v5 v5.10.0/go.mod
h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle v0.0.0-20190413234325-e4ced69a3a2b/go.mod
h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod
h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.0/go.mod
h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.1/go.mod
h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.3/go.mod
h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
-github.com/jackc/puddle/v2 v2.2.1
h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk=
-github.com/jackc/puddle/v2 v2.2.1/go.mod
h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
+github.com/jackc/puddle/v2 v2.2.2
h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
+github.com/jackc/puddle/v2 v2.2.2/go.mod
h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod
h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/jinzhu/inflection v1.0.0
h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
@@ -376,8 +376,6 @@ github.com/lib/pq v1.0.0/go.mod
h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
-github.com/lib/pq v1.10.2 h1:AqzbZs4ZoCBp+GtejcpCpcxM3zlSMx29dXbUSeVtJb8=
-github.com/lib/pq v1.10.2/go.mod
h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/libgit2/git2go/v34 v34.0.0
h1:UKoUaKLmiCRbOCD3PtUi2hD6hESSXzME/9OUZrGcgu8=
github.com/libgit2/git2go/v34 v34.0.0/go.mod
h1:blVco2jDAw6YTXkErMMqzHLcAjKkwF0aWIRHBqiJkZ0=
github.com/magiconair/properties v1.8.5
h1:b6kJs+EmPFMYGkow9GiUyCyOvIwYetYJ3fSaWak/Gls=
@@ -501,8 +499,8 @@ github.com/stretchr/testify v1.7.1/go.mod
h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.8.0/go.mod
h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod
h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.3/go.mod
h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
-github.com/stretchr/testify v1.9.0
h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
-github.com/stretchr/testify v1.9.0/go.mod
h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/stretchr/testify v1.11.1
h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod
h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.2.0
h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod
h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/swaggo/files v1.0.1 h1:J1bVJ4XHZNq0I46UU90611i9/YzdrF7x92oX1ig5IdE=
@@ -573,8 +571,8 @@ golang.org/x/crypto
v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5y
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod
h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod
h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4=
golang.org/x/crypto v0.7.0/go.mod
h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
-golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw=
-golang.org/x/crypto v0.26.0/go.mod
h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54=
+golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
+golang.org/x/crypto v0.41.0/go.mod
h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod
h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod
h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod
h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -614,8 +612,8 @@ golang.org/x/mod v0.4.1/go.mod
h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod
h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=
-golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
+golang.org/x/mod v0.27.0 h1:kb+q2PyFnEADO2IEF935ehFUXlWiNjJWtRNgBLSfbxQ=
+golang.org/x/mod v0.27.0/go.mod h1:rWI627Fq0DEoudcK+MBkNkCe0EetEaDSwJJkCcjpazc=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod
h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod
h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod
h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -660,8 +658,8 @@ golang.org/x/net v0.2.0/go.mod
h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
-golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
-golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
+golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
+golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod
h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod
h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod
h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -689,8 +687,8 @@ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod
h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod
h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod
h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ=
-golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
+golang.org/x/sync v0.17.0/go.mod
h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod
h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod
h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod
h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -753,16 +751,16 @@ golang.org/x/sys v0.2.0/go.mod
h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.23.0 h1:YfKFowiIMvtgl1UERQoTPPToxltDeZfbj4H7dVUCwmM=
-golang.org/x/sys v0.23.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
+golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod
h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod
h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod
h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
-golang.org/x/term v0.23.0 h1:F6D4vR+EHoL9/sWAWgAR1H2DcHr4PareCbAaCo1RpuU=
-golang.org/x/term v0.23.0/go.mod
h1:DgV24QBUrK6jhZXl+20l6UWznPlwAHm1Q1mGHtydmSk=
+golang.org/x/term v0.34.0 h1:O/2T7POpk0ZZ7MAzMeWFSg6S5IpWd/RXDlM9hgM3DR4=
+golang.org/x/term v0.34.0/go.mod
h1:5jC53AEywhIVebHgPVeg0mj8OD3VO9OzclacVrqpaAw=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod
h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod
h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -776,8 +774,8 @@ golang.org/x/text v0.3.8/go.mod
h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc=
-golang.org/x/text v0.17.0/go.mod
h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
+golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
+golang.org/x/text v0.29.0/go.mod
h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod
h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod
h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod
h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -838,8 +836,8 @@ golang.org/x/tools v0.1.0/go.mod
h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.2/go.mod
h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.12/go.mod
h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod
h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d
h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=
-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod
h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
+golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
+golang.org/x/tools v0.36.0/go.mod
h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
golang.org/x/xerrors v0.0.0-20190410155217-1f06c39b4373/go.mod
h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod
h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod
h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
diff --git a/backend/plugins/starrocks/tasks/tasks.go
b/backend/plugins/starrocks/tasks/tasks.go
index 79fbe5d2a..8245ad538 100644
--- a/backend/plugins/starrocks/tasks/tasks.go
+++ b/backend/plugins/starrocks/tasks/tasks.go
@@ -37,7 +37,7 @@ import (
"github.com/apache/incubator-devlake/impls/dalgorm"
"github.com/apache/incubator-devlake/plugins/starrocks/utils"
- "github.com/lib/pq"
+ "github.com/jackc/pgx/v5/pgtype"
"gorm.io/driver/mysql"
"gorm.io/driver/postgres"
"gorm.io/gorm"
@@ -306,6 +306,9 @@ func copyDataToDst(dc *DataConfigParams, columnMap
map[string]string, orderBy st
}
var batchCount int
+ // typeMap registers the default pgtype catalog, so it is built once
per query
+ // and reused for every array column of every row.
+ typeMap := pgtype.NewMap()
for rows.Next() {
select {
case <-c.GetContext().Done():
@@ -320,7 +323,7 @@ func copyDataToDst(dc *DataConfigParams, columnMap
map[string]string, orderBy st
if strings.HasPrefix(dataType, "array") {
var arr []string
columns[i] = &arr
- columnPointers[i] = pq.Array(&arr)
+ columnPointers[i] = typeMap.SQLScanner(&arr)
} else {
columnPointers[i] = &columns[i]
}
diff --git a/backend/plugins/starrocks/tasks/tasks_test.go
b/backend/plugins/starrocks/tasks/tasks_test.go
new file mode 100644
index 000000000..18b53c063
--- /dev/null
+++ b/backend/plugins/starrocks/tasks/tasks_test.go
@@ -0,0 +1,68 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package tasks
+
+import (
+ "testing"
+
+ "github.com/jackc/pgx/v5/pgtype"
+ "github.com/stretchr/testify/assert"
+)
+
+// TestArrayColumnScan documents the behaviour of the scanner used for
PostgreSQL
+// array columns in export(). The PostgreSQL driver hands array columns to
+// database/sql as the raw array literal (a string), so the destination has to
be
+// an sql.Scanner able to parse that literal.
+func TestArrayColumnScan(t *testing.T) {
+ cases := []struct {
+ name string
+ literal string
+ expected []string
+ }{
+ {"simple", `{a,b}`, []string{"a", "b"}},
+ {"empty", `{}`, []string{}},
+ {"single empty element", `{""}`, []string{""}},
+ {"quoted separators", `{"a,x","b\"y"}`, []string{"a,x", `b"y`}},
+ {"utf8 and spaces", `{ä ö,"x y"}`, []string{"ä ö", "x y"}},
+ }
+
+ typeMap := pgtype.NewMap()
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ var arr []string
+ scanner := typeMap.SQLScanner(&arr)
+
+ assert.NoError(t, scanner.Scan(tc.literal))
+ assert.Equal(t, tc.expected, arr)
+
+ // database/sql may deliver the very same literal as
[]byte.
+ var fromBytes []string
+ assert.NoError(t,
typeMap.SQLScanner(&fromBytes).Scan([]byte(tc.literal)))
+ assert.Equal(t, tc.expected, fromBytes)
+ })
+ }
+}
+
+// TestArrayColumnScanNullElement pins down that NULL elements are still
rejected:
+// the previous implementation could not handle them either, so this is not a
+// behaviour change.
+func TestArrayColumnScanNullElement(t *testing.T) {
+ var arr []string
+ err := pgtype.NewMap().SQLScanner(&arr).Scan(`{NULL}`)
+ assert.Error(t, err)
+}
diff --git a/backend/scripts/install-go.sh b/backend/scripts/install-go.sh
new file mode 100755
index 000000000..15a44c20c
--- /dev/null
+++ b/backend/scripts/install-go.sh
@@ -0,0 +1,94 @@
+#!/bin/sh
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eu
+
+GO_VERSION=1.26.2
+INSTALL_DIR=/opt/go/$GO_VERSION
+GO_BIN=$INSTALL_DIR/bin/go
+
+get_version() {
+ "$1" version 2>/dev/null |
+ sed -n 's/^go version go\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\)
.*/\1/p'
+}
+
+use_installed_go() {
+ export PATH="$INSTALL_DIR/bin:$PATH"
+ if [ -n "${GITHUB_PATH:-}" ]; then
+ echo "$INSTALL_DIR/bin" >> "$GITHUB_PATH"
+ fi
+}
+
+current_go=$(command -v go 2>/dev/null || true)
+if [ -n "$current_go" ]; then
+ current_version=$(get_version "$current_go" || true)
+ if [ "$current_version" = "$GO_VERSION" ]; then
+ echo "Go $current_version is already installed"
+ exit 0
+ fi
+fi
+
+if [ -x "$GO_BIN" ]; then
+ installed_version=$(get_version "$GO_BIN" || true)
+ if [ "$installed_version" = "$GO_VERSION" ]; then
+ use_installed_go
+ echo "Go $installed_version is already installed in $INSTALL_DIR"
+ exit 0
+ fi
+fi
+
+case "$(uname -m)" in
+ x86_64|amd64)
+ archive_name=go${GO_VERSION}.linux-amd64.tar.gz
+
archive_sha256=990e6b4bbba816dc3ee129eaeaf4b42f17c2800b88a2166c265ac1a200262282
+ ;;
+ aarch64|arm64)
+ archive_name=go${GO_VERSION}.linux-arm64.tar.gz
+
archive_sha256=c958a1fe1b361391db163a485e21f5f228142d6f8b584f6bef89b26f66dc5b23
+ ;;
+ *)
+ echo "unsupported architecture: $(uname -m)" >&2
+ exit 1
+ ;;
+esac
+
+if [ "$(id -u)" -ne 0 ]; then
+ echo "install-go.sh must run as root" >&2
+ exit 1
+fi
+
+download_url=https://go.dev/dl/$archive_name
+work_dir=$(mktemp -d)
+trap 'rm -rf "$work_dir"' EXIT HUP INT TERM
+
+archive=$work_dir/$archive_name
+curl --fail --location --retry 3 "$download_url" --output "$archive"
+echo "$archive_sha256 $archive" | sha256sum --check --strict
+
+tar -xzf "$archive" -C "$work_dir"
+mkdir -p "$(dirname "$INSTALL_DIR")"
+rm -rf "$INSTALL_DIR"
+mv "$work_dir/go" "$INSTALL_DIR"
+
+use_installed_go
+installed_version=$(get_version "$GO_BIN")
+if [ "$installed_version" != "$GO_VERSION" ]; then
+ echo "expected Go $GO_VERSION, found ${installed_version:-unknown}" >&2
+ exit 1
+fi
+
+echo "installed Go $installed_version in $INSTALL_DIR"
