[
https://issues.apache.org/jira/browse/MRESOLVER-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17629481#comment-17629481
]
ASF GitHub Bot commented on MRESOLVER-283:
------------------------------------------
michael-o commented on code in PR #213:
URL: https://github.com/apache/maven-resolver/pull/213#discussion_r1014828015
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/DefaultMetadataResolver.java:
##########
@@ -374,41 +391,36 @@ else if ( exception == null )
if ( !tasks.isEmpty() )
{
- int threads = ConfigUtils.getInteger( session, 4,
CONFIG_PROP_THREADS );
- Executor executor = getExecutor( Math.min( tasks.size(), threads )
);
- try
+ RunnableErrorForwarder errorForwarder = new
RunnableErrorForwarder();
+ ArrayList<Runnable> runnable = new ArrayList<>( tasks.size() );
Review Comment:
runnables
##########
maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java:
##########
@@ -146,8 +150,10 @@
this.repository = repository;
this.fileProcessor = fileProcessor;
this.providedChecksumsSources = providedChecksumsSources;
+ this.resolverExecutor = resolverExecutorService.getResolverExecutor(
session, RepositoryConnector.class,
+ ConfigUtils.getInteger( session, CONFIG_PROP_THREADS_DEFAULT,
CONFIG_PROP_THREADS,
+ "maven.artifact.threads" ) );
Review Comment:
We should also deprecate this property since it does not reflect reality. MD
is not artifacts, but this connector does both.
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutorService.java:
##########
@@ -0,0 +1,92 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+import org.eclipse.aether.spi.concurrency.ResolverExecutorService;
+import org.eclipse.aether.util.concurrency.WorkerThreadFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * This implementation uses {@link RepositorySystemSession#getData()} to store
created {@link ExecutorService}
+ * instances. It creates instances that may be eventually garbage collected,
so no explicit shutdown happens on
+ * them. When {@code maxThreads} parameter is 1 (accepted values are greater
than zero), this implementation assumes
+ * caller wants "direct execution" (on caller thread) and creates {@link
ResolverExecutor} instances accordingly.
+ */
+@Singleton
+@Named
+public final class DefaultResolverExecutorService implements
ResolverExecutorService
+{
+ @Override
+ public ResolverExecutor getResolverExecutor( RepositorySystemSession
session,
+ Class<?> service,
+ int maxThreads )
+ {
+ requireNonNull( session );
+ requireNonNull( service );
+ if ( maxThreads < 1 )
+ {
+ throw new IllegalArgumentException( "threads must be greater than
zero" );
+ }
+
+ final ExecutorService executorService;
+ if ( maxThreads == 1 ) // direct
+ {
+ executorService = null;
+ }
+ else // shared && pooled
+ {
+ String key = DefaultResolverExecutorService.class.getName() + "."
+ service.getSimpleName();
+ executorService = (ExecutorService) session.getData()
+ .computeIfAbsent( key, () -> createExecutorService(
service, maxThreads ) );
+ }
+ return new DefaultResolverExecutor( executorService );
+ }
+
+ /**
+ * Creates am {@link ExecutorService} that allows its core threads to die
off in case of inactivity, and allows
+ * for proper garbage collection. This is important detail, as these
instances are kept within session data, and
+ * currently there is no way to shut down them.
+ */
+ private ExecutorService createExecutorService( Class<?> service, int
maxThreads )
+ {
+ ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
+ maxThreads,
+ maxThreads,
+ 3L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(),
+ new WorkerThreadFactory( getClass().getSimpleName() + "-" +
service.getSimpleName() + "-" )
+ );
+ threadPoolExecutor.allowCoreThreadTimeOut( true );
Review Comment:
Why is this necessary?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutorService.java:
##########
@@ -0,0 +1,92 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+import org.eclipse.aether.spi.concurrency.ResolverExecutorService;
+import org.eclipse.aether.util.concurrency.WorkerThreadFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * This implementation uses {@link RepositorySystemSession#getData()} to store
created {@link ExecutorService}
+ * instances. It creates instances that may be eventually garbage collected,
so no explicit shutdown happens on
+ * them. When {@code maxThreads} parameter is 1 (accepted values are greater
than zero), this implementation assumes
+ * caller wants "direct execution" (on caller thread) and creates {@link
ResolverExecutor} instances accordingly.
+ */
+@Singleton
+@Named
+public final class DefaultResolverExecutorService implements
ResolverExecutorService
+{
+ @Override
+ public ResolverExecutor getResolverExecutor( RepositorySystemSession
session,
+ Class<?> service,
+ int maxThreads )
+ {
+ requireNonNull( session );
+ requireNonNull( service );
+ if ( maxThreads < 1 )
+ {
+ throw new IllegalArgumentException( "threads must be greater than
zero" );
+ }
+
+ final ExecutorService executorService;
+ if ( maxThreads == 1 ) // direct
+ {
+ executorService = null;
+ }
+ else // shared && pooled
+ {
+ String key = DefaultResolverExecutorService.class.getName() + "."
+ service.getSimpleName();
+ executorService = (ExecutorService) session.getData()
+ .computeIfAbsent( key, () -> createExecutorService(
service, maxThreads ) );
+ }
+ return new DefaultResolverExecutor( executorService );
+ }
+
+ /**
+ * Creates am {@link ExecutorService} that allows its core threads to die
off in case of inactivity, and allows
+ * for proper garbage collection. This is important detail, as these
instances are kept within session data, and
+ * currently there is no way to shut down them.
+ */
+ private ExecutorService createExecutorService( Class<?> service, int
maxThreads )
+ {
+ ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
+ maxThreads,
+ maxThreads,
+ 3L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(),
+ new WorkerThreadFactory( getClass().getSimpleName() + "-" +
service.getSimpleName() + "-" )
Review Comment:
So we have lost the per repo thread pool?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/DefaultMetadataResolver.java:
##########
@@ -83,8 +78,16 @@
implements MetadataResolver, Service
{
+ /**
+ * The count of threads to be used when resolving metadata in parallel,
default value 4.
+ */
private static final String CONFIG_PROP_THREADS =
"aether.metadataResolver.threads";
+ /**
+ * The default value for {@link #CONFIG_PROP_THREADS}.
+ */
+ private static final int CONFIG_PROP_THREADS_DEFAULT = 4;
Review Comment:
Why not 5 like in the other one?
##########
maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java:
##########
@@ -287,9 +269,10 @@ public void get( Collection<? extends ArtifactDownload>
artifactDownloads,
task = new GetTaskRunner( location, transfer.getFile(),
checksumPolicy,
checksumAlgorithmFactories, checksumLocations,
providedChecksums, listener );
}
- executor.execute( errorForwarder.wrap( task ) );
+ runnable.add( errorForwarder.wrap( task ) );
}
+ resolverExecutor.submitOrDirect( runnable );
Review Comment:
Why wait until they are all collected instead separate MD and A like before?
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutorService.java:
##########
@@ -0,0 +1,92 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import javax.inject.Named;
+import javax.inject.Singleton;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.aether.RepositorySystemSession;
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+import org.eclipse.aether.spi.concurrency.ResolverExecutorService;
+import org.eclipse.aether.util.concurrency.WorkerThreadFactory;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * This implementation uses {@link RepositorySystemSession#getData()} to store
created {@link ExecutorService}
+ * instances. It creates instances that may be eventually garbage collected,
so no explicit shutdown happens on
+ * them. When {@code maxThreads} parameter is 1 (accepted values are greater
than zero), this implementation assumes
+ * caller wants "direct execution" (on caller thread) and creates {@link
ResolverExecutor} instances accordingly.
+ */
+@Singleton
+@Named
+public final class DefaultResolverExecutorService implements
ResolverExecutorService
+{
+ @Override
+ public ResolverExecutor getResolverExecutor( RepositorySystemSession
session,
+ Class<?> service,
+ int maxThreads )
+ {
+ requireNonNull( session );
+ requireNonNull( service );
+ if ( maxThreads < 1 )
+ {
+ throw new IllegalArgumentException( "threads must be greater than
zero" );
Review Comment:
maxThreads must be...
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/concurrency/DefaultResolverExecutor.java:
##########
@@ -0,0 +1,93 @@
+package org.eclipse.aether.internal.impl.concurrency;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collection;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.eclipse.aether.spi.concurrency.ResolverExecutor;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Default implementation of {@link ResolverExecutor}.
+ * <p>
+ * It relies on ctor passed {@link ExecutorService} that may be {@code null},
in which case "direct invocation" (on
+ * caller thread) happens, otherwise the non-null executor service is used.
+ */
+final class DefaultResolverExecutor implements ResolverExecutor
+{
+ private final ExecutorService executorService;
+
+ DefaultResolverExecutor( final ExecutorService executorService )
+ {
+ this.executorService = executorService;
+ }
+
+ @Override
+ public void submitOrDirect( Collection<Runnable> tasks )
Review Comment:
I think from an abstract PoV just `submit` is enough. It is an
implementation detail whether it should be executed directly or scheduled for
execution. The behavior should be described in the Javadoc of the class.
##########
maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java:
##########
@@ -225,11 +205,13 @@ public void get( Collection<? extends ArtifactDownload>
artifactDownloads,
throw new IllegalStateException( "connector closed" );
}
- Executor executor = getExecutor( artifactDownloads, metadataDownloads
);
RunnableErrorForwarder errorForwarder = new RunnableErrorForwarder();
List<ChecksumAlgorithmFactory> checksumAlgorithmFactories =
layout.getChecksumAlgorithmFactories();
+ Collection<? extends MetadataDownload> mds = safe( metadataDownloads );
+ Collection<? extends ArtifactDownload> ads = safe( artifactDownloads );
+ ArrayList<Runnable> runnable = new ArrayList<>( mds.size() +
ads.size() );
Review Comment:
runnables
> Introduce resolver wide "shared" executor service
> -------------------------------------------------
>
> Key: MRESOLVER-283
> URL: https://issues.apache.org/jira/browse/MRESOLVER-283
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
> Reporter: Tamas Cservenak
> Assignee: Tamas Cservenak
> Priority: Major
> Fix For: 1.9.0
>
>
> More and more component in resolver does parallel processing (BF collector,
> MD resolver, basic connector), and they all create, maintain their own
> executor instance.
> Instead of this, create one shared service component and just reuse it
> accross resolver.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
