wmedvede commented on code in PR #2790: URL: https://github.com/apache/incubator-kie-tools/pull/2790#discussion_r1916795669
########## packages/sonataflow-operator/operator.yaml: ########## @@ -19403,9 +19422,14 @@ spec: for the workflow maxProperties: 2 properties: - migrateDBOnStartUp: - description: Whether to migrate database on service startup? - type: boolean + dbMigrationStrategy: Review Comment: I think that if we are not implementing the dbMigrationStrategy for the workflows, we shouldn't add it in the SonataFlow CRD. ########## packages/sonataflow-operator/config/manifests/bases/sonataflow-operator.clusterserviceversion.yaml: ########## Review Comment: I think the dbMigrationStrategy must be added to the DI and JS in this file too. dbMigrationStrategy: default: service description: |- DB Migration approach for service? job: use job based approach service: service itself shall migrate the db none: no db migration needed ########## packages/sonataflow-operator/config/manager/controllers_cfg.yaml: ########## @@ -30,6 +30,8 @@ jobsServiceEphemeralImageTag: "docker.io/apache/incubator-kie-kogito-jobs-servic # The Data Index image to use, if empty the operator will use the default Apache Community one based on the current operator's version dataIndexPostgreSQLImageTag: "docker.io/apache/incubator-kie-kogito-data-index-postgresql:main" dataIndexEphemeralImageTag: "docker.io/apache/incubator-kie-kogito-data-index-ephemeral:main" +# The Kogito PostgreSQL DB Migrator image to use (TBD: to replace with apache image) +kogitoDBMigratorToolImageTag: "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" Review Comment: For the other DI and JS imageTags, we just say, `jobsServicePostgreSQLImageTag`, `jobsServiceEphemeralImageTag`, `dataIndexPostgreSQLImageTag`, `dataIndexEphemeralImageTag` and we don't prefix with kogito, even when they are. I think we should just say: `dbMigratorToolImageTag` for consistency. And propagate the change along the controller config struct, etc. wdyt? @ricardozanini @rhkp ########## packages/sonataflow-operator/internal/controller/cfg/controllers_cfg.go: ########## @@ -63,6 +63,7 @@ type ControllersCfg struct { JobsServiceEphemeralImageTag string `yaml:"jobsServiceEphemeralImageTag,omitempty"` DataIndexPostgreSQLImageTag string `yaml:"dataIndexPostgreSQLImageTag,omitempty"` DataIndexEphemeralImageTag string `yaml:"dataIndexEphemeralImageTag,omitempty"` + KogitoDBMigratorToolImageTag string `yaml:"kogitoDBMigratorToolImageTag,omitempty"` Review Comment: see comments on this field name ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/sonataflow-platform/01-sonataflow_platform.yaml: ########## @@ -0,0 +1,43 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong license format ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/pg-service/kustomization.yaml: ########## @@ -0,0 +1,30 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong licence format ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" Review Comment: Unused constant ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## Review Comment: This file needs tests, the same as we have for other ones. ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { + if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef != nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 { + return persistencePostgreSQL.ServiceRef.DatabaseSchema + } + + if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) > 0 { + jdbcURL := persistencePostgreSQL.JdbcUrl + _, a, found := strings.Cut(jdbcURL, "currentSchema=") + + if found { + if strings.Contains(a, "&") { + b, _, found := strings.Cut(a, "&") + if found { + return b + } + } else { + return a + } + } + } + return defaultSchemaName +} + +func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { + databaseSchema := defaultDBSchemaName + dataSourcePort := constants.DefaultPostgreSQLPort + databaseName := constants.DefaultDatabaseName + postgresServiceName := constants.DefaultPostgresServiceName + + dataSourceURL := "" + + if postgresql.ServiceRef != nil { + if len(postgresql.ServiceRef.DatabaseSchema) > 0 { + databaseSchema = postgresql.ServiceRef.DatabaseSchema + } + if postgresql.ServiceRef.Port != nil { + dataSourcePort = *postgresql.ServiceRef.Port + } + if len(postgresql.ServiceRef.DatabaseName) > 0 { + databaseName = postgresql.ServiceRef.DatabaseName + } + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema) + } else if len(postgresql.JdbcUrl) > 0 { + dataSourceURL = postgresql.JdbcUrl + } else { + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName, dataSourcePort, databaseName, databaseSchema) + } + + return dataSourceURL +} + +func getQuarkusDataSourceFromPersistence(ctx context.Context, platform *operatorapi.SonataFlowPlatform, persistence *operatorapi.PersistenceOptionsSpec, defaultSchemaName string) *QuarkusDataSource { + if persistence != nil && persistence.PostgreSQL != nil { + quarkusDataSource := &QuarkusDataSource{} + quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL, defaultSchemaName) + quarkusDataSource.Username, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace) + quarkusDataSource.Password, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace) + quarkusDataSource.Schema = getDBSchemaName(persistence.PostgreSQL, defaultSchemaName) + return quarkusDataSource + } + + return nil +} + +func NewDBMigratorJobData(ctx context.Context, client client.Client, platform *operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS services.PlatformServiceHandler) *DBMigratorJob { + + diJobsBasedDBMigration := false + jsJobsBasedDBMigration := false + + if pshDI.IsPersistenceEnabledtInSpec() { + diJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence) + } + if pshJS.IsPersistenceEnabledtInSpec() { + jsJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence) + } + + if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) || (pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) { + quarkusDataSourceDataIndex := &QuarkusDataSource{} + quarkusDataSourceJobService := &QuarkusDataSource{} + + if diJobsBasedDBMigration { + quarkusDataSourceDataIndex = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.DataIndex.Persistence, "data-index-service") Review Comment: Also, to get the default schema name, we do https://github.com/apache/incubator-kie-tools/blob/ab197e99ea446da684baa2a533a68413e2e23e22/packages/sonataflow-operator/internal/controller/platform/services/services.go#L291 I think we should do something similar here instead of setting "data-index-service" ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { + if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef != nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 { + return persistencePostgreSQL.ServiceRef.DatabaseSchema + } + + if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) > 0 { + jdbcURL := persistencePostgreSQL.JdbcUrl + _, a, found := strings.Cut(jdbcURL, "currentSchema=") + + if found { + if strings.Contains(a, "&") { + b, _, found := strings.Cut(a, "&") + if found { + return b + } + } else { + return a + } + } + } + return defaultSchemaName +} + +func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { + databaseSchema := defaultDBSchemaName + dataSourcePort := constants.DefaultPostgreSQLPort + databaseName := constants.DefaultDatabaseName + postgresServiceName := constants.DefaultPostgresServiceName + + dataSourceURL := "" + + if postgresql.ServiceRef != nil { + if len(postgresql.ServiceRef.DatabaseSchema) > 0 { + databaseSchema = postgresql.ServiceRef.DatabaseSchema + } + if postgresql.ServiceRef.Port != nil { + dataSourcePort = *postgresql.ServiceRef.Port + } + if len(postgresql.ServiceRef.DatabaseName) > 0 { + databaseName = postgresql.ServiceRef.DatabaseName + } + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema) + } else if len(postgresql.JdbcUrl) > 0 { + dataSourceURL = postgresql.JdbcUrl + } else { + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName, dataSourcePort, databaseName, databaseSchema) + } + + return dataSourceURL +} + +func getQuarkusDataSourceFromPersistence(ctx context.Context, platform *operatorapi.SonataFlowPlatform, persistence *operatorapi.PersistenceOptionsSpec, defaultSchemaName string) *QuarkusDataSource { + if persistence != nil && persistence.PostgreSQL != nil { + quarkusDataSource := &QuarkusDataSource{} + quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL, defaultSchemaName) + quarkusDataSource.Username, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace) + quarkusDataSource.Password, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace) + quarkusDataSource.Schema = getDBSchemaName(persistence.PostgreSQL, defaultSchemaName) + return quarkusDataSource + } + + return nil +} + +func NewDBMigratorJobData(ctx context.Context, client client.Client, platform *operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS services.PlatformServiceHandler) *DBMigratorJob { + + diJobsBasedDBMigration := false + jsJobsBasedDBMigration := false + + if pshDI.IsPersistenceEnabledtInSpec() { + diJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence) + } + if pshJS.IsPersistenceEnabledtInSpec() { + jsJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence) + } + + if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) || (pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) { + quarkusDataSourceDataIndex := &QuarkusDataSource{} + quarkusDataSourceJobService := &QuarkusDataSource{} + + if diJobsBasedDBMigration { + quarkusDataSourceDataIndex = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.DataIndex.Persistence, "data-index-service") + } + + if jsJobsBasedDBMigration { + quarkusDataSourceJobService = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.JobService.Persistence, "jobs-service") Review Comment: and same comment for default schema name calculation, see: j.GetServiceName() ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/pg-service/01-postgres.yaml: ########## @@ -0,0 +1,85 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong licence format ########## packages/sonataflow-operator/internal/controller/platform/services/secrets.go: ########## @@ -0,0 +1,38 @@ +// Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong licence header, we don't use the yearly format. ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/job_based_db_migration/sonataflow-platform/kustomization.yaml: ########## @@ -0,0 +1,19 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong license format ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/service_based_db_migration/01-postgres.yaml: ########## @@ -0,0 +1,85 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong license format ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/service_based_db_migration/02-sonataflow_platform.yaml: ########## @@ -0,0 +1,65 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong license format ########## packages/sonataflow-operator/test/e2e/testdata/platform/persistence/service_based_db_migration/kustomization.yaml: ########## @@ -0,0 +1,31 @@ +# Copyright 2024 Apache Software Foundation (ASF) Review Comment: wrong license format ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { Review Comment: tests please ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { + if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef != nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 { + return persistencePostgreSQL.ServiceRef.DatabaseSchema + } + + if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) > 0 { + jdbcURL := persistencePostgreSQL.JdbcUrl + _, a, found := strings.Cut(jdbcURL, "currentSchema=") + + if found { + if strings.Contains(a, "&") { + b, _, found := strings.Cut(a, "&") + if found { + return b + } + } else { + return a + } + } + } + return defaultSchemaName +} + +func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { Review Comment: I believe this calculations are kind of duplicated here: https://github.com/apache/incubator-kie-tools/blob/c690d462602da4cf2c01b90279815b60a1959162/packages/sonataflow-operator/internal/controller/profiles/common/persistence/postgresql.go#L44 I think we should explore moving this calculation postgresql.go and reuse. ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { + if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef != nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 { + return persistencePostgreSQL.ServiceRef.DatabaseSchema + } + + if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) > 0 { + jdbcURL := persistencePostgreSQL.JdbcUrl + _, a, found := strings.Cut(jdbcURL, "currentSchema=") + + if found { + if strings.Contains(a, "&") { + b, _, found := strings.Cut(a, "&") + if found { + return b + } + } else { + return a + } + } + } + return defaultSchemaName +} + +func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { + databaseSchema := defaultDBSchemaName + dataSourcePort := constants.DefaultPostgreSQLPort + databaseName := constants.DefaultDatabaseName + postgresServiceName := constants.DefaultPostgresServiceName + + dataSourceURL := "" + + if postgresql.ServiceRef != nil { + if len(postgresql.ServiceRef.DatabaseSchema) > 0 { + databaseSchema = postgresql.ServiceRef.DatabaseSchema + } + if postgresql.ServiceRef.Port != nil { + dataSourcePort = *postgresql.ServiceRef.Port + } + if len(postgresql.ServiceRef.DatabaseName) > 0 { + databaseName = postgresql.ServiceRef.DatabaseName + } + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema) + } else if len(postgresql.JdbcUrl) > 0 { + dataSourceURL = postgresql.JdbcUrl + } else { + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName, dataSourcePort, databaseName, databaseSchema) + } + + return dataSourceURL +} + +func getQuarkusDataSourceFromPersistence(ctx context.Context, platform *operatorapi.SonataFlowPlatform, persistence *operatorapi.PersistenceOptionsSpec, defaultSchemaName string) *QuarkusDataSource { + if persistence != nil && persistence.PostgreSQL != nil { + quarkusDataSource := &QuarkusDataSource{} + quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL, defaultSchemaName) + quarkusDataSource.Username, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace) + quarkusDataSource.Password, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace) + quarkusDataSource.Schema = getDBSchemaName(persistence.PostgreSQL, defaultSchemaName) + return quarkusDataSource + } + + return nil +} + +func NewDBMigratorJobData(ctx context.Context, client client.Client, platform *operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS services.PlatformServiceHandler) *DBMigratorJob { + + diJobsBasedDBMigration := false + jsJobsBasedDBMigration := false + + if pshDI.IsPersistenceEnabledtInSpec() { + diJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence) + } + if pshJS.IsPersistenceEnabledtInSpec() { + jsJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence) + } + + if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) || (pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) { + quarkusDataSourceDataIndex := &QuarkusDataSource{} + quarkusDataSourceJobService := &QuarkusDataSource{} + + if diJobsBasedDBMigration { + quarkusDataSourceDataIndex = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.DataIndex.Persistence, "data-index-service") + } + + if jsJobsBasedDBMigration { + quarkusDataSourceJobService = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.JobService.Persistence, "jobs-service") Review Comment: same as for the DI, see comment above. ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { + if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef != nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 { + return persistencePostgreSQL.ServiceRef.DatabaseSchema + } + + if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) > 0 { + jdbcURL := persistencePostgreSQL.JdbcUrl + _, a, found := strings.Cut(jdbcURL, "currentSchema=") + + if found { + if strings.Contains(a, "&") { + b, _, found := strings.Cut(a, "&") + if found { + return b + } + } else { + return a + } + } + } + return defaultSchemaName +} + +func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { + databaseSchema := defaultDBSchemaName + dataSourcePort := constants.DefaultPostgreSQLPort + databaseName := constants.DefaultDatabaseName + postgresServiceName := constants.DefaultPostgresServiceName + + dataSourceURL := "" + + if postgresql.ServiceRef != nil { + if len(postgresql.ServiceRef.DatabaseSchema) > 0 { + databaseSchema = postgresql.ServiceRef.DatabaseSchema + } + if postgresql.ServiceRef.Port != nil { + dataSourcePort = *postgresql.ServiceRef.Port + } + if len(postgresql.ServiceRef.DatabaseName) > 0 { + databaseName = postgresql.ServiceRef.DatabaseName + } + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema) + } else if len(postgresql.JdbcUrl) > 0 { + dataSourceURL = postgresql.JdbcUrl + } else { + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName, dataSourcePort, databaseName, databaseSchema) + } + + return dataSourceURL +} + +func getQuarkusDataSourceFromPersistence(ctx context.Context, platform *operatorapi.SonataFlowPlatform, persistence *operatorapi.PersistenceOptionsSpec, defaultSchemaName string) *QuarkusDataSource { + if persistence != nil && persistence.PostgreSQL != nil { + quarkusDataSource := &QuarkusDataSource{} + quarkusDataSource.JdbcUrl = getJdbcUrl(persistence.PostgreSQL, defaultSchemaName) + quarkusDataSource.Username, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.UserKey, platform.Namespace) + quarkusDataSource.Password, _ = services.GetSecretKeyValueString(ctx, persistence.PostgreSQL.SecretRef.Name, persistence.PostgreSQL.SecretRef.PasswordKey, platform.Namespace) + quarkusDataSource.Schema = getDBSchemaName(persistence.PostgreSQL, defaultSchemaName) + return quarkusDataSource + } + + return nil +} + +func NewDBMigratorJobData(ctx context.Context, client client.Client, platform *operatorapi.SonataFlowPlatform, pshDI services.PlatformServiceHandler, pshJS services.PlatformServiceHandler) *DBMigratorJob { + + diJobsBasedDBMigration := false + jsJobsBasedDBMigration := false + + if pshDI.IsPersistenceEnabledtInSpec() { + diJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.DataIndex.Persistence) + } + if pshJS.IsPersistenceEnabledtInSpec() { + jsJobsBasedDBMigration = services.IsJobsBasedDBMigration(platform.Spec.Services.JobService.Persistence) + } + + if (pshDI.IsServiceSetInSpec() && diJobsBasedDBMigration) || (pshJS.IsServiceSetInSpec() && jsJobsBasedDBMigration) { + quarkusDataSourceDataIndex := &QuarkusDataSource{} + quarkusDataSourceJobService := &QuarkusDataSource{} + + if diJobsBasedDBMigration { + quarkusDataSourceDataIndex = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.DataIndex.Persistence, "data-index-service") Review Comment: When get the DataSource, we must consider the comments above, i.e., the potential case where the persistence is defined at the SPF level. In the file packages/sonataflow-operator/internal/controller/profiles/common/persistence/postgresql.go I think we have some methods that can be useful for these calculations. ########## packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go: ########## @@ -0,0 +1,356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package platform + +import ( + "context" + "errors" + "fmt" + "strconv" + "strings" + + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/pointer" + + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/version" + + operatorapi "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/api/v1alpha08" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/container-builder/client" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/cfg" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/platform/services" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/internal/controller/profiles/common/constants" + "github.com/apache/incubator-kie-tools/packages/sonataflow-operator/log" +) + +type QuarkusDataSource struct { + JdbcUrl string + Username string + Password string + Schema string +} + +type DBMigratorJob struct { + MigrateDBDataIndex bool + DataIndexDataSource *QuarkusDataSource + MigrateDBJobsService bool + JobsServiceDataSource *QuarkusDataSource +} + +const ( + dbMigrationJobName = "sonataflow-db-migrator-job" + dbMigrationContainerName = "db-migration-container" + dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" + dbMigrationCmd = "./migration.sh" + dbMigrationJobFailed = 1 + dbMigrationJobSucceeded = 1 + + migrateDBDataIndex = "MIGRATE_DB_DATAINDEX" + quarkusDataSourceDataIndexJdbcURL = "QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL" + quarkusDataSourceDataIndexUserName = "QUARKUS_DATASOURCE_DATAINDEX_USERNAME" + quarkusDataSourceDataIndexPassword = "QUARKUS_DATASOURCE_DATAINDEX_PASSWORD" + quarkusFlywayDataIndexSchemas = "QUARKUS_FLYWAY_DATAINDEX_SCHEMAS" + + migrateDBJobsService = "MIGRATE_DB_JOBSSERVICE" + quarkusDataSourceJobsServiceJdbcURL = "QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL" + quarkusDataSourceJobsServiceUserName = "QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME" + quarkusDataSourceJobsServicePassword = "QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD" + quarkusFlywayJobsServiceSchemas = "QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS" +) + +type DBMigrationJobCfg struct { + JobName string + ContainerName string + ToolImageName string + MigrationCmd string +} + +func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { + if persistencePostgreSQL != nil && persistencePostgreSQL.ServiceRef != nil && len(persistencePostgreSQL.ServiceRef.DatabaseSchema) > 0 { + return persistencePostgreSQL.ServiceRef.DatabaseSchema + } + + if persistencePostgreSQL != nil && len(persistencePostgreSQL.JdbcUrl) > 0 { + jdbcURL := persistencePostgreSQL.JdbcUrl + _, a, found := strings.Cut(jdbcURL, "currentSchema=") + + if found { + if strings.Contains(a, "&") { + b, _, found := strings.Cut(a, "&") + if found { + return b + } + } else { + return a + } + } + } + return defaultSchemaName +} + +func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { + databaseSchema := defaultDBSchemaName + dataSourcePort := constants.DefaultPostgreSQLPort + databaseName := constants.DefaultDatabaseName + postgresServiceName := constants.DefaultPostgresServiceName + + dataSourceURL := "" + + if postgresql.ServiceRef != nil { + if len(postgresql.ServiceRef.DatabaseSchema) > 0 { + databaseSchema = postgresql.ServiceRef.DatabaseSchema + } + if postgresql.ServiceRef.Port != nil { + dataSourcePort = *postgresql.ServiceRef.Port + } + if len(postgresql.ServiceRef.DatabaseName) > 0 { + databaseName = postgresql.ServiceRef.DatabaseName + } + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresql.ServiceRef.Name, dataSourcePort, databaseName, databaseSchema) + } else if len(postgresql.JdbcUrl) > 0 { + dataSourceURL = postgresql.JdbcUrl + } else { + dataSourceURL = fmt.Sprintf("jdbc:postgresql://%s:%d/%s?currentSchema=%s", postgresServiceName, dataSourcePort, databaseName, databaseSchema) + } + + return dataSourceURL +} + +func getQuarkusDataSourceFromPersistence(ctx context.Context, platform *operatorapi.SonataFlowPlatform, persistence *operatorapi.PersistenceOptionsSpec, defaultSchemaName string) *QuarkusDataSource { Review Comment: Important: Persistence configuration for a service can be taken not only from eg. spec.services.jobsService but also from the SPF global persistence configuration. see: https://sonataflow.org/serverlessworkflow/main/cloud/operator/supporting-services.html#common-persistence-configuration https://sonataflow.org/serverlessworkflow/main/cloud/operator/using-persistence.html#configuring-persistence-using-the-sonataflowplatform-cr Along this PR, we must consider that potential case too. I.e, the service don't have "persistence", but the SPF do. Examples: ``` apiVersion: sonataflow.org/v1alpha08 kind: SonataFlowPlatform metadata: name: sonataflow-platform spec: services: # no persistence is defined for the DI, it will take it from the SFP if any. # and apply the by default DBMigrationStrategy dataIndex: enabled: true jobService: # The only persistence setting we do is the DBMigrationStrategy # So, the service grabs the persistence configuation from the SPF and the given DBMigrationStrategy # from here. enabled: true persistence: dbMigrationStrategy: "job" # global SPF persitence configuration. persistence: postgresql: secretRef: name: postgres-secrets userKey: POSTGRESQL_USER passwordKey: POSTGRESQL_PASSWORD serviceRef: name: postgres databaseName: sonataflow ``` @rhkp take look please and propagate in all places where needed. @ricardozanini see my considerations for this particular case, where a service takes the persistence from the SPF, but we still can mark at the service level the dbMigrationStrategy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@kie.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@kie.apache.org For additional commands, e-mail: commits-h...@kie.apache.org