wmedvede commented on code in PR #2790:
URL:
https://github.com/apache/incubator-kie-tools/pull/2790#discussion_r1916795669
##########
packages/sonataflow-operator/operator.yaml:
##########
@@ -19403,9 +19422,14 @@ spec:
for the workflow
maxProperties: 2
properties:
- migrateDBOnStartUp:
- description: Whether to migrate database on service
startup?
- type: boolean
+ dbMigrationStrategy:
Review Comment:
I think that if we are not implementing the dbMigrationStrategy for the
workflows, we shouldn't add it in the SonataFlow CRD.
##########
packages/sonataflow-operator/config/manifests/bases/sonataflow-operator.clusterserviceversion.yaml:
##########
Review Comment:
I think the dbMigrationStrategy must be added to the DI and JS in this file
too.
dbMigrationStrategy:
default: service
description: |-
DB Migration approach for service?
job: use job based approach
service: service itself shall migrate the db
none: no db migration needed
##########
packages/sonataflow-operator/config/manager/controllers_cfg.yaml:
##########
@@ -30,6 +30,8 @@ jobsServiceEphemeralImageTag:
"docker.io/apache/incubator-kie-kogito-jobs-servic
# The Data Index image to use, if empty the operator will use the default
Apache Community one based on the current operator's version
dataIndexPostgreSQLImageTag:
"docker.io/apache/incubator-kie-kogito-data-index-postgresql:main"
dataIndexEphemeralImageTag:
"docker.io/apache/incubator-kie-kogito-data-index-ephemeral:main"
+# The Kogito PostgreSQL DB Migrator image to use (TBD: to replace with apache
image)
+kogitoDBMigratorToolImageTag:
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
Review Comment:
For the other DI and JS imageTags, we just say,
`jobsServicePostgreSQLImageTag`, `jobsServiceEphemeralImageTag`,
`dataIndexPostgreSQLImageTag`, `dataIndexEphemeralImageTag` and we don't prefix
with kogito, even when they are.
I think we should just say: `dbMigratorToolImageTag` for consistency.
And propagate the change along the controller config struct, etc.
wdyt? @ricardozanini @rhkp
##########
packages/sonataflow-operator/internal/controller/cfg/controllers_cfg.go:
##########
@@ -63,6 +63,7 @@ type ControllersCfg struct {
JobsServiceEphemeralImageTag string
`yaml:"jobsServiceEphemeralImageTag,omitempty"`
DataIndexPostgreSQLImageTag string
`yaml:"dataIndexPostgreSQLImageTag,omitempty"`
DataIndexEphemeralImageTag string
`yaml:"dataIndexEphemeralImageTag,omitempty"`
+ KogitoDBMigratorToolImageTag string
`yaml:"kogitoDBMigratorToolImageTag,omitempty"`
Review Comment:
see comments on this field name
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/sonataflow-platform/01-sonataflow_platform.yaml:
##########
@@ -0,0 +1,43 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong license format
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/pg-service/kustomization.yaml:
##########
@@ -0,0 +1,30 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong licence format
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
Review Comment:
Unused constant
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
Review Comment:
This file needs tests, the same as we have for other ones.
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
+ if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef !=
nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 {
+ return persistencePostgreSQL.ServiceRef.DatabaseSchema
+ }
+
+ if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) >
0 {
+ jdbcURL := persistencePostgreSQL.JdbcUrl
+ _, a, found := strings.Cut(jdbcURL, "currentSchema=")
+
+ if found {
+ if strings.Contains(a, "&") {
+ b, _, found := strings.Cut(a, "&")
+ if found {
+ return b
+ }
+ } else {
+ return a
+ }
+ }
+ }
+ return defaultSchemaName
+}
+
+func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL,
defaultDBSchemaName string) string {
+ databaseSchema := defaultDBSchemaName
+ dataSourcePort := constants.DefaultPostgreSQLPort
+ databaseName := constants.DefaultDatabaseName
+ postgresServiceName := constants.DefaultPostgresServiceName
+
+ dataSourceURL := ""
+
+ if postgresql.ServiceRef != nil {
+ if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+ databaseSchema = postgresql.ServiceRef.DatabaseSchema
+ }
+ if postgresql.ServiceRef.Port != nil {
+ dataSourcePort = *postgresql.ServiceRef.Port
+ }
+ if len(postgresql.ServiceRef.DatabaseName) > 0 {
+ databaseName = postgresql.ServiceRef.DatabaseName
+ }
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s",
postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema)
+ } else if len(postgresql.JdbcUrl) > 0 {
+ dataSourceURL = postgresql.JdbcUrl
+ } else {
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName,
dataSourcePort, databaseName, databaseSchema)
+ }
+
+ return dataSourceURL
+}
+
+func getQuarkusDataSourceFromPersistence(ctx context.Context, platform
*operatorapi.SonataFlowPlatform, persistence
*operatorapi.PersistenceOptionsSpec, defaultSchemaName string)
*QuarkusDataSource {
+ if persistence != nil && persistence.PostgreSQL != nil {
+ quarkusDataSource := &QuarkusDataSource{}
+ quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL,
defaultSchemaName)
+ quarkusDataSource.Username, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace)
+ quarkusDataSource.Password, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace)
+ quarkusDataSource.Schema =
getDBSchemaName(persistence.PostgreSQL, defaultSchemaName)
+ return quarkusDataSource
+ }
+
+ return nil
+}
+
+func NewDBMigratorJobData(ctx context.Context, client client.Client, platform
*operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS
services.PlatformServiceHandler) *DBMigratorJob {
+
+ diJobsBasedDBMigration := false
+ jsJobsBasedDBMigration := false
+
+ if pshDI.IsPersistenceEnabledtInSpec() {
+ diJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence)
+ }
+ if pshJS.IsPersistenceEnabledtInSpec() {
+ jsJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence)
+ }
+
+ if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) ||
(pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) {
+ quarkusDataSourceDataIndex := &QuarkusDataSource{}
+ quarkusDataSourceJobService := &QuarkusDataSource{}
+
+ if diJobsBasedDBMigration {
+ quarkusDataSourceDataIndex =
getQuarkusDataSourceFromPersistence(ctx, platform,
platform.Spec.Services.DataIndex.Persistence, "data-index-service")
Review Comment:
Also, to get the default schema name, we do
https://github.com/apache/incubator-kie-tools/blob/ab197e99ea446da684baa2a533a68413e2e23e22/packages/sonataflow-operator/internal/controller/platform/services/services.go#L291
I think we should do something similar here instead of setting
"data-index-service"
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
+ if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef !=
nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 {
+ return persistencePostgreSQL.ServiceRef.DatabaseSchema
+ }
+
+ if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) >
0 {
+ jdbcURL := persistencePostgreSQL.JdbcUrl
+ _, a, found := strings.Cut(jdbcURL, "currentSchema=")
+
+ if found {
+ if strings.Contains(a, "&") {
+ b, _, found := strings.Cut(a, "&")
+ if found {
+ return b
+ }
+ } else {
+ return a
+ }
+ }
+ }
+ return defaultSchemaName
+}
+
+func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL,
defaultDBSchemaName string) string {
+ databaseSchema := defaultDBSchemaName
+ dataSourcePort := constants.DefaultPostgreSQLPort
+ databaseName := constants.DefaultDatabaseName
+ postgresServiceName := constants.DefaultPostgresServiceName
+
+ dataSourceURL := ""
+
+ if postgresql.ServiceRef != nil {
+ if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+ databaseSchema = postgresql.ServiceRef.DatabaseSchema
+ }
+ if postgresql.ServiceRef.Port != nil {
+ dataSourcePort = *postgresql.ServiceRef.Port
+ }
+ if len(postgresql.ServiceRef.DatabaseName) > 0 {
+ databaseName = postgresql.ServiceRef.DatabaseName
+ }
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s",
postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema)
+ } else if len(postgresql.JdbcUrl) > 0 {
+ dataSourceURL = postgresql.JdbcUrl
+ } else {
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName,
dataSourcePort, databaseName, databaseSchema)
+ }
+
+ return dataSourceURL
+}
+
+func getQuarkusDataSourceFromPersistence(ctx context.Context, platform
*operatorapi.SonataFlowPlatform, persistence
*operatorapi.PersistenceOptionsSpec, defaultSchemaName string)
*QuarkusDataSource {
+ if persistence != nil && persistence.PostgreSQL != nil {
+ quarkusDataSource := &QuarkusDataSource{}
+ quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL,
defaultSchemaName)
+ quarkusDataSource.Username, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace)
+ quarkusDataSource.Password, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace)
+ quarkusDataSource.Schema =
getDBSchemaName(persistence.PostgreSQL, defaultSchemaName)
+ return quarkusDataSource
+ }
+
+ return nil
+}
+
+func NewDBMigratorJobData(ctx context.Context, client client.Client, platform
*operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS
services.PlatformServiceHandler) *DBMigratorJob {
+
+ diJobsBasedDBMigration := false
+ jsJobsBasedDBMigration := false
+
+ if pshDI.IsPersistenceEnabledtInSpec() {
+ diJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence)
+ }
+ if pshJS.IsPersistenceEnabledtInSpec() {
+ jsJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence)
+ }
+
+ if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) ||
(pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) {
+ quarkusDataSourceDataIndex := &QuarkusDataSource{}
+ quarkusDataSourceJobService := &QuarkusDataSource{}
+
+ if diJobsBasedDBMigration {
+ quarkusDataSourceDataIndex =
getQuarkusDataSourceFromPersistence(ctx, platform,
platform.Spec.Services.DataIndex.Persistence, "data-index-service")
+ }
+
+ if jsJobsBasedDBMigration {
+ quarkusDataSourceJobService =
getQuarkusDataSourceFromPersistence(ctx, platform,
platform.Spec.Services.JobService.Persistence, "jobs-service")
Review Comment:
and same comment for default schema name calculation, see:
j.GetServiceName()
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/pg-service/01-postgres.yaml:
##########
@@ -0,0 +1,85 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong licence format
##########
packages/sonataflow-operator/internal/controller/platform/services/secrets.go:
##########
@@ -0,0 +1,38 @@
+// Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong licence header, we don't use the yearly format.
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/sonataflow-platform/kustomization.yaml:
##########
@@ -0,0 +1,19 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong license format
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/service_based_db_migration/01-postgres.yaml:
##########
@@ -0,0 +1,85 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong license format
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/service_based_db_migration/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,65 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong license format
##########
packages/sonataflow-operator/test/e2e/testdata/platform/persistence/service_based_db_migration/kustomization.yaml:
##########
@@ -0,0 +1,31 @@
+# Copyright 2024 Apache Software Foundation (ASF)
Review Comment:
wrong license format
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
Review Comment:
tests please
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
+ if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef !=
nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 {
+ return persistencePostgreSQL.ServiceRef.DatabaseSchema
+ }
+
+ if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) >
0 {
+ jdbcURL := persistencePostgreSQL.JdbcUrl
+ _, a, found := strings.Cut(jdbcURL, "currentSchema=")
+
+ if found {
+ if strings.Contains(a, "&") {
+ b, _, found := strings.Cut(a, "&")
+ if found {
+ return b
+ }
+ } else {
+ return a
+ }
+ }
+ }
+ return defaultSchemaName
+}
+
+func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL,
defaultDBSchemaName string) string {
Review Comment:
I believe this calculations are kind of duplicated here:
https://github.com/apache/incubator-kie-tools/blob/c690d462602da4cf2c01b90279815b60a1959162/packages/sonataflow-operator/internal/controller/profiles/common/persistence/postgresql.go#L44
I think we should explore moving this calculation postgresql.go and reuse.
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
+ if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef !=
nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 {
+ return persistencePostgreSQL.ServiceRef.DatabaseSchema
+ }
+
+ if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) >
0 {
+ jdbcURL := persistencePostgreSQL.JdbcUrl
+ _, a, found := strings.Cut(jdbcURL, "currentSchema=")
+
+ if found {
+ if strings.Contains(a, "&") {
+ b, _, found := strings.Cut(a, "&")
+ if found {
+ return b
+ }
+ } else {
+ return a
+ }
+ }
+ }
+ return defaultSchemaName
+}
+
+func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL,
defaultDBSchemaName string) string {
+ databaseSchema := defaultDBSchemaName
+ dataSourcePort := constants.DefaultPostgreSQLPort
+ databaseName := constants.DefaultDatabaseName
+ postgresServiceName := constants.DefaultPostgresServiceName
+
+ dataSourceURL := ""
+
+ if postgresql.ServiceRef != nil {
+ if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+ databaseSchema = postgresql.ServiceRef.DatabaseSchema
+ }
+ if postgresql.ServiceRef.Port != nil {
+ dataSourcePort = *postgresql.ServiceRef.Port
+ }
+ if len(postgresql.ServiceRef.DatabaseName) > 0 {
+ databaseName = postgresql.ServiceRef.DatabaseName
+ }
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s",
postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema)
+ } else if len(postgresql.JdbcUrl) > 0 {
+ dataSourceURL = postgresql.JdbcUrl
+ } else {
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName,
dataSourcePort, databaseName, databaseSchema)
+ }
+
+ return dataSourceURL
+}
+
+func getQuarkusDataSourceFromPersistence(ctx context.Context, platform
*operatorapi.SonataFlowPlatform, persistence
*operatorapi.PersistenceOptionsSpec, defaultSchemaName string)
*QuarkusDataSource {
+ if persistence != nil && persistence.PostgreSQL != nil {
+ quarkusDataSource := &QuarkusDataSource{}
+ quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL,
defaultSchemaName)
+ quarkusDataSource.Username, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace)
+ quarkusDataSource.Password, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace)
+ quarkusDataSource.Schema =
getDBSchemaName(persistence.PostgreSQL, defaultSchemaName)
+ return quarkusDataSource
+ }
+
+ return nil
+}
+
+func NewDBMigratorJobData(ctx context.Context, client client.Client, platform
*operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS
services.PlatformServiceHandler) *DBMigratorJob {
+
+ diJobsBasedDBMigration := false
+ jsJobsBasedDBMigration := false
+
+ if pshDI.IsPersistenceEnabledtInSpec() {
+ diJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence)
+ }
+ if pshJS.IsPersistenceEnabledtInSpec() {
+ jsJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence)
+ }
+
+ if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) ||
(pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) {
+ quarkusDataSourceDataIndex := &QuarkusDataSource{}
+ quarkusDataSourceJobService := &QuarkusDataSource{}
+
+ if diJobsBasedDBMigration {
+ quarkusDataSourceDataIndex =
getQuarkusDataSourceFromPersistence(ctx, platform,
platform.Spec.Services.DataIndex.Persistence, "data-index-service")
+ }
+
+ if jsJobsBasedDBMigration {
+ quarkusDataSourceJobService =
getQuarkusDataSourceFromPersistence(ctx, platform,
platform.Spec.Services.JobService.Persistence, "jobs-service")
Review Comment:
same as for the DI, see comment above.
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
+ if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef !=
nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 {
+ return persistencePostgreSQL.ServiceRef.DatabaseSchema
+ }
+
+ if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) >
0 {
+ jdbcURL := persistencePostgreSQL.JdbcUrl
+ _, a, found := strings.Cut(jdbcURL, "currentSchema=")
+
+ if found {
+ if strings.Contains(a, "&") {
+ b, _, found := strings.Cut(a, "&")
+ if found {
+ return b
+ }
+ } else {
+ return a
+ }
+ }
+ }
+ return defaultSchemaName
+}
+
+func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL,
defaultDBSchemaName string) string {
+ databaseSchema := defaultDBSchemaName
+ dataSourcePort := constants.DefaultPostgreSQLPort
+ databaseName := constants.DefaultDatabaseName
+ postgresServiceName := constants.DefaultPostgresServiceName
+
+ dataSourceURL := ""
+
+ if postgresql.ServiceRef != nil {
+ if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+ databaseSchema = postgresql.ServiceRef.DatabaseSchema
+ }
+ if postgresql.ServiceRef.Port != nil {
+ dataSourcePort = *postgresql.ServiceRef.Port
+ }
+ if len(postgresql.ServiceRef.DatabaseName) > 0 {
+ databaseName = postgresql.ServiceRef.DatabaseName
+ }
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s",
postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema)
+ } else if len(postgresql.JdbcUrl) > 0 {
+ dataSourceURL = postgresql.JdbcUrl
+ } else {
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName,
dataSourcePort, databaseName, databaseSchema)
+ }
+
+ return dataSourceURL
+}
+
+func getQuarkusDataSourceFromPersistence(ctx context.Context, platform
*operatorapi.SonataFlowPlatform, persistence
*operatorapi.PersistenceOptionsSpec, defaultSchemaName string)
*QuarkusDataSource {
+ if persistence != nil && persistence.PostgreSQL != nil {
+ quarkusDataSource := &QuarkusDataSource{}
+ quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL,
defaultSchemaName)
+ quarkusDataSource.Username, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace)
+ quarkusDataSource.Password, _ =
services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name,
persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace)
+ quarkusDataSource.Schema =
getDBSchemaName(persistence.PostgreSQL, defaultSchemaName)
+ return quarkusDataSource
+ }
+
+ return nil
+}
+
+func NewDBMigratorJobData(ctx context.Context, client client.Client, platform
*operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS
services.PlatformServiceHandler) *DBMigratorJob {
+
+ diJobsBasedDBMigration := false
+ jsJobsBasedDBMigration := false
+
+ if pshDI.IsPersistenceEnabledtInSpec() {
+ diJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence)
+ }
+ if pshJS.IsPersistenceEnabledtInSpec() {
+ jsJobsBasedDBMigration =
services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence)
+ }
+
+ if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) ||
(pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) {
+ quarkusDataSourceDataIndex := &QuarkusDataSource{}
+ quarkusDataSourceJobService := &QuarkusDataSource{}
+
+ if diJobsBasedDBMigration {
+ quarkusDataSourceDataIndex =
getQuarkusDataSourceFromPersistence(ctx, platform,
platform.Spec.Services.DataIndex.Persistence, "data-index-service")
Review Comment:
When get the DataSource, we must consider the comments above, i.e., the
potential case where the persistence is defined at the SPF level.
In the file
packages/sonataflow-operator/internal/controller/profiles/common/persistence/postgresql.go
I think we have some methods that can be useful for these calculations.
##########
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package platform
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ batchv1 "k8s.io/api/batch/v1"
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/klog/v2"
+ "k8s.io/utils/pointer"
+
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version"
+
+ operatorapi
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services"
+
"github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants"
+ "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log"
+)
+
+type QuarkusDataSource struct {
+ JdbcUrl string
+ Username string
+ Password string
+ Schema string
+}
+
+type DBMigratorJob struct {
+ MigrateDBDataIndex bool
+ DataIndexDataSource *QuarkusDataSource
+ MigrateDBJobsService bool
+ JobsServiceDataSource *QuarkusDataSource
+}
+
+const (
+ dbMigrationJobName = "sonataflow-db-migrator-job"
+ dbMigrationContainerName = "db-migration-container"
+ dbMigratorToolImage =
"quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest"
+ dbMigrationCmd = "./migration.sh"
+ dbMigrationJobFailed = 1
+ dbMigrationJobSucceeded = 1
+
+ migrateDBDataIndex = "MIGRATE_DB_DATAINDEX"
+ quarkusDataSourceDataIndexJdbcURL =
"QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL"
+ quarkusDataSourceDataIndexUserName =
"QUARKUS_DATASOURCE_DATAINDEX_USERNAME"
+ quarkusDataSourceDataIndexPassword =
"QUARKUS_DATASOURCE_DATAINDEX_PASSWORD"
+ quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS"
+
+ migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE"
+ quarkusDataSourceJobsServiceJdbcURL =
"QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL"
+ quarkusDataSourceJobsServiceUserName =
"QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME"
+ quarkusDataSourceJobsServicePassword =
"QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD"
+ quarkusFlywayJobsServiceSchemas =
"QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS"
+)
+
+type DBMigrationJobCfg struct {
+ JobName string
+ ContainerName string
+ ToolImageName string
+ MigrationCmd string
+}
+
+func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL,
defaultSchemaName string) string {
+ if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef !=
nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 {
+ return persistencePostgreSQL.ServiceRef.DatabaseSchema
+ }
+
+ if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) >
0 {
+ jdbcURL := persistencePostgreSQL.JdbcUrl
+ _, a, found := strings.Cut(jdbcURL, "currentSchema=")
+
+ if found {
+ if strings.Contains(a, "&") {
+ b, _, found := strings.Cut(a, "&")
+ if found {
+ return b
+ }
+ } else {
+ return a
+ }
+ }
+ }
+ return defaultSchemaName
+}
+
+func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL,
defaultDBSchemaName string) string {
+ databaseSchema := defaultDBSchemaName
+ dataSourcePort := constants.DefaultPostgreSQLPort
+ databaseName := constants.DefaultDatabaseName
+ postgresServiceName := constants.DefaultPostgresServiceName
+
+ dataSourceURL := ""
+
+ if postgresql.ServiceRef != nil {
+ if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+ databaseSchema = postgresql.ServiceRef.DatabaseSchema
+ }
+ if postgresql.ServiceRef.Port != nil {
+ dataSourcePort = *postgresql.ServiceRef.Port
+ }
+ if len(postgresql.ServiceRef.DatabaseName) > 0 {
+ databaseName = postgresql.ServiceRef.DatabaseName
+ }
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s",
postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema)
+ } else if len(postgresql.JdbcUrl) > 0 {
+ dataSourceURL = postgresql.JdbcUrl
+ } else {
+ dataSourceURL =
fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName,
dataSourcePort, databaseName, databaseSchema)
+ }
+
+ return dataSourceURL
+}
+
+func getQuarkusDataSourceFromPersistence(ctx context.Context, platform
*operatorapi.SonataFlowPlatform, persistence
*operatorapi.PersistenceOptionsSpec, defaultSchemaName string)
*QuarkusDataSource {
Review Comment:
Important:
Persistence configuration for a service can be taken not only from eg.
spec.services.jobsService but also from the SPF global persistence
configuration.
see:
https://sonataflow.org/serverlessworkflow/main/cloud/operator/supporting-services.html#common-persistence-configuration
https://sonataflow.org/serverlessworkflow/main/cloud/operator/using-persistence.html#configuring-persistence-using-the-sonataflowplatform-cr
Along this PR, we must consider that potential case too.
I.e, the service don't have "persistence", but the SPF do.
Examples:
```
apiVersion: sonataflow.org/v1alpha08
kind: SonataFlowPlatform
metadata:
name: sonataflow-platform
spec:
services:
# no persistence is defined for the DI, it will take it from the
SFP if any.
# and apply the by default DBMigrationStrategy
dataIndex:
enabled: true
jobService:
# The only persistence setting we do is the DBMigrationStrategy
# So, the service grabs the persistence configuation from the SPF
and the given DBMigrationStrategy
# from here.
enabled: true
persistence:
dbMigrationStrategy: "job"
# global SPF persitence configuration.
persistence:
postgresql:
secretRef:
name: postgres-secrets
userKey: POSTGRESQL_USER
passwordKey: POSTGRESQL_PASSWORD
serviceRef:
name: postgres
databaseName: sonataflow
```
@rhkp take look please and propagate in all places where needed.
@ricardozanini see my considerations for this particular case, where a
service takes the persistence from the SPF, but we still can mark at the
service level the dbMigrationStrategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
