oscerd commented on code in PR #30191:
URL: https://github.com/apache/camel/pull/30191#discussion_r4589202961
##########
test-infra/camel-test-infra-google-common/src/main/java/org/apache/camel/test/infra/google/common/services/FlociGcpContainer.java:
##########
@@ -47,4 +76,86 @@ public FlociGcpContainer(String imageName, boolean
fixedPort) {
public String serviceAddress() {
return String.format("http://%s:%d", getHost(),
getMappedPort(SERVICE_PORT));
}
+
+ /**
+ * Lets Floci GCP start containers of its own, such as the SQL engine of
BigQuery: mounts the Docker socket of the
+ * host, and gives the containers that Floci GCP starts a resource
namespace of this container, by which
+ * {@link #stop()} removes them. Each container has a namespace of its
own: when it starts, Floci GCP removes the
+ * containers of its namespace that it finds, as left over.
+ */
+ public FlociGcpContainer withDockerSocket() {
+ resourceNamespace = "camel" +
UUID.randomUUID().toString().replace("-", "").substring(0, 12);
+ withEnv(RESOURCE_NAMESPACE_ENV, resourceNamespace);
+ return
withCreateContainerCmdModifier(FlociGcpContainer::bindDockerSocket);
+ }
+
+ /**
+ * Has Floci GCP run the queries of BigQuery on the floci-duck image of
+ * {@link GoogleProperties#BIGQUERY_ENGINE_CONTAINER}, which this method
pulls as Testcontainers pulls its own
+ * images, so that image substitution and registry authentication apply:
Floci GCP would pull it itself otherwise.
+ * Needs {@link #withDockerSocket()}.
+ */
+ public FlociGcpContainer withBigQueryEngine() {
Review Comment:
Done in 9b740b0: `withBigQueryEngine()` now throws `IllegalStateException:
withDockerSocket() must come before withBigQueryEngine(): Floci GCP starts the
engine through the Docker socket` when the socket was not mounted.
_Claude Code on behalf of Andrea Cosentino_
##########
components/camel-google/camel-google-bigquery/src/test/java/org/apache/camel/component/google/bigquery/integration/BigQueryITSupport.java:
##########
@@ -99,9 +106,7 @@ protected void addBigquerySqlComponent(CamelContext context)
{
GoogleBigQuerySQLConfiguration configuration = new
GoogleBigQuerySQLConfiguration()
.setServiceAccountKey(CREDENTIALS_FILE_LOCATION);
- connectionFactory = new GoogleBigQueryConnectionFactory()
- .setServiceAccountKeyFile(configuration.getServiceAccountKey())
- .setCamelContext(context);
+ connectionFactory = createConnectionFactory(context);
Review Comment:
Done in 9b740b0: `createCamelContext()` builds one connection factory, and
both components take it. Locally it is `new
GoogleBigQueryConnectionFactory(client)` with the Floci GCP client, so the
subclass is gone too.
_Claude Code on behalf of Andrea Cosentino_
##########
test-infra/camel-test-infra-google-bigquery/src/main/java/org/apache/camel/test/infra/google/bigquery/services/GoogleBigQueryLocalContainerInfraService.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel.test.infra.google.bigquery.services;
+
+import org.apache.camel.spi.annotations.InfraService;
+import org.apache.camel.test.infra.common.LocalPropertyResolver;
+import org.apache.camel.test.infra.common.services.ContainerEnvironmentUtil;
+import org.apache.camel.test.infra.common.services.ContainerService;
+import
org.apache.camel.test.infra.google.bigquery.common.GoogleBigQueryProperties;
+import org.apache.camel.test.infra.google.common.GoogleProperties;
+import org.apache.camel.test.infra.google.common.services.FlociGcpContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A local BigQuery service on Floci GCP, which runs the queries on
floci-duck, a SQL engine that it starts as a
+ * container of its own through the Docker socket. Floci GCP accepts any
credentials, and gives an access token to a
+ * service account of any key at {@code <endpoint>/token}.
+ */
+@InfraService(service = GoogleBigQueryInfraService.class,
+ description = "Local Google BigQuery service with Floci GCP",
+ serviceAlias = "google", serviceImplementationAlias = "bigquery")
+public class GoogleBigQueryLocalContainerInfraService
+ implements GoogleBigQueryInfraService,
ContainerService<FlociGcpContainer> {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoogleBigQueryLocalContainerInfraService.class);
+ private static final String DEFAULT_PROJECT_ID = "test-project";
+
+ private final FlociGcpContainer container;
+
+ public GoogleBigQueryLocalContainerInfraService() {
+ this(LocalPropertyResolver.getProperty(FlociGcpContainer.class,
GoogleProperties.GOOGLE_CONTAINER));
+ }
+
+ public GoogleBigQueryLocalContainerInfraService(String imageName) {
+ container = initContainer(imageName);
+ String name = ContainerEnvironmentUtil.containerName(this.getClass());
+ if (name != null) {
+ container.withCreateContainerCmdModifier(cmd ->
cmd.withName(name));
+ }
+ }
+
+ public GoogleBigQueryLocalContainerInfraService(FlociGcpContainer
container) {
+ this.container = container;
+ }
+
+ protected FlociGcpContainer initContainer(String imageName) {
+ return new FlociGcpContainer(imageName,
ContainerEnvironmentUtil.isFixedPort(this.getClass()))
+ .withDockerSocket()
+ .withBigQueryEngine();
Review Comment:
Thanks for reproducing it. Done in 9b740b0, as you suggested: the local
service only mounts the socket in `initContainer()`, and calls
`withBigQueryEngine()` in `initialize()`, before `start()`. Nothing talks to
Docker while the test class is loaded any more.
Checked with the Docker socket hidden from the build (`bwrap --dev-bind / /
--tmpfs /run`): 9 skipped, BUILD SUCCESS. With Docker: 7 pass and the 2 of
`InsertIdIT` are skipped.
_Claude Code on behalf of Andrea Cosentino_
##########
test-infra/camel-test-infra-google-common/src/main/java/org/apache/camel/test/infra/google/common/services/FlociGcpContainer.java:
##########
@@ -47,4 +76,86 @@ public FlociGcpContainer(String imageName, boolean
fixedPort) {
public String serviceAddress() {
return String.format("http://%s:%d", getHost(),
getMappedPort(SERVICE_PORT));
}
+
+ /**
+ * Lets Floci GCP start containers of its own, such as the SQL engine of
BigQuery: mounts the Docker socket of the
+ * host, and gives the containers that Floci GCP starts a resource
namespace of this container, by which
+ * {@link #stop()} removes them. Each container has a namespace of its
own: when it starts, Floci GCP removes the
+ * containers of its namespace that it finds, as left over.
+ */
+ public FlociGcpContainer withDockerSocket() {
+ resourceNamespace = "camel" +
UUID.randomUUID().toString().replace("-", "").substring(0, 12);
+ withEnv(RESOURCE_NAMESPACE_ENV, resourceNamespace);
+ return
withCreateContainerCmdModifier(FlociGcpContainer::bindDockerSocket);
+ }
+
+ /**
+ * Has Floci GCP run the queries of BigQuery on the floci-duck image of
+ * {@link GoogleProperties#BIGQUERY_ENGINE_CONTAINER}, which this method
pulls as Testcontainers pulls its own
+ * images, so that image substitution and registry authentication apply:
Floci GCP would pull it itself otherwise.
+ * Needs {@link #withDockerSocket()}.
+ */
+ public FlociGcpContainer withBigQueryEngine() {
+ String image =
LocalPropertyResolver.getProperty(FlociGcpContainer.class,
+ GoogleProperties.BIGQUERY_ENGINE_CONTAINER);
+ return withEnv(BIGQUERY_ENGINE_IMAGE_ENV, new
RemoteDockerImage(DockerImageName.parse(image)).get());
+ }
+
+ @Override
+ public void start() {
+ if (resourceNamespace != null) {
+ // Testcontainers does not know the containers that Floci GCP
starts: have them removed when the JVM exits,
+ // should this container not be stopped
+
ResourceReaper.instance().registerLabelsFilterForCleanup(Map.of(RESOURCE_NAMESPACE_LABEL,
resourceNamespace));
+ }
+ super.start();
+ }
+
+ @Override
+ public void stop() {
+ try {
+ super.stop();
+ } finally {
+ if (resourceNamespace != null) {
+ removeStartedContainers();
+ }
+ }
+ }
+
+ // the containers that Floci GCP started are not removed with it
+ private void removeStartedContainers() {
+ DockerClient client = DockerClientFactory.instance().client();
+ List<Container> containers = client.listContainersCmd()
+ .withShowAll(true)
+ .withLabelFilter(Map.of(RESOURCE_NAMESPACE_LABEL,
resourceNamespace))
+ .exec();
+ for (Container container : containers) {
+ LOG.debug("Removing the container {} that Floci GCP started",
Arrays.toString(container.getNames()));
+ try {
+
client.removeContainerCmd(container.getId()).withForce(true).withRemoveVolumes(true).exec();
Review Comment:
Done in 9b740b0: `removeStartedContainers()` now catches `ConflictException`
next to `NotFoundException`, so a removal that another stop already started is
ignored. I applied the same change to `FlociAzContainer`, which has the same
code. The root cause, one shutdown hook per test class, is CAMEL-26762.
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]