JiayaoS commented on code in PR #23837: URL: https://github.com/apache/kafka/pull/23837#discussion_r4300367142
########## core/src/test/java/kafka/docker/NativeImageConfigTest.java: ########## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package kafka.docker; + +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.server.config.AbstractKafkaConfig; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; + +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TreeSet; + +import io.github.classgraph.ClassGraph; +import io.github.classgraph.ClassInfo; +import io.github.classgraph.ScanResult; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * The native image can only create the classes registered in reflect-config.json by reflection, and a missing + * entry fails at runtime rather than at build time. See KAFKA-19583, KAFKA-19584 and KAFKA-21045. + */ +public class NativeImageConfigTest { + + private static final Path REFLECT_CONFIG = Path.of(System.getProperty("user.dir"), + "..", "docker", "native", "native-image-configs", "reflect-config.json"); + + private static final String HOW_TO_FIX = "Register their no-arg constructors in " + REFLECT_CONFIG + + ", or instantiate them without reflection (see KAFKA-20825)."; + + // Interfaces whose implementations may be instantiated reflectively by class name. + // To find more, search for Utils.newInstance and getConfiguredInstance(s). + private static final List<String> PLUGGABLE_INTERFACES = List.of( + "javax.security.auth.spi.LoginModule", + "javax.security.sasl.SaslClientFactory", + "javax.security.sasl.SaslServerFactory", + "kafka.metrics.KafkaMetricsReporter", + "org.apache.kafka.common.config.provider.ConfigProvider", + "org.apache.kafka.common.metrics.MetricsReporter", + "org.apache.kafka.common.replica.ReplicaSelector", + "org.apache.kafka.common.security.auth.AuthenticateCallbackHandler", + "org.apache.kafka.common.security.auth.KafkaPrincipalBuilder", + "org.apache.kafka.common.security.auth.Login", + "org.apache.kafka.common.security.auth.SecurityProviderCreator", + "org.apache.kafka.common.security.auth.SslEngineFactory", + "org.apache.kafka.common.security.oauthbearer.JwtRetriever", + "org.apache.kafka.common.security.oauthbearer.JwtValidator", + "org.apache.kafka.coordinator.group.api.assignor.ConsumerGroupPartitionAssignor", + "org.apache.kafka.coordinator.group.api.assignor.ShareGroupPartitionAssignor", + "org.apache.kafka.coordinator.group.api.streams.StreamsGroupTopologyDescriptionPlugin", + "org.apache.kafka.coordinator.group.api.streams.assignor.TaskAssignor", + "org.apache.kafka.coordinator.group.streams.AssignmentRefiner", + "org.apache.kafka.server.authorizer.Authorizer", + "org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager", + "org.apache.kafka.server.log.remote.storage.RemoteStorageManager", + "org.apache.kafka.server.policy.AlterConfigPolicy", + "org.apache.kafka.server.policy.CreateTopicPolicy", + "org.apache.kafka.server.quota.ClientQuotaCallback" + ); Review Comment: > I think, hardcoded `PLUGGABLE_INTERFACES` list has same weakness as a bug it guards against. For isntance if someone adds a new pluggable interface buit forgets to put it here then its implementation are never checked. Could we somehow derive these interfaces dynamically? > > It reminds me how we treated the versioning in MetadataVersion class. Thanks, that's a fair point. The list has the same weakness as the bug it guards against. I looked into a few approaches to derive the interfaces automatically, and none of them was complete: - Common super type: the pluggable interfaces don't share one. - ConfigDef: only CLASS-type configs can be reliably detected, and ConfigDef doesn't record the interface they expect. Plugin configs of type STRING or LIST (e.g. authorizer.class.name, config.providers) look the same as ordinary string or list configs, so they can't be detected by type. - Call sites: I also considered scanning the bytecode for getConfiguredInstance/Utils.newInstance calls. A simple scan would not cover all paths: some pass Class variables that require further type analysis, while JAAS/SASL implementations are loaded by the JDK. Covering these cases would require more involved analysis. Unlike MetadataVersion, there doesn't seem to be a single source of truth to iterate over. I may well be missing something, though. Do you have any suggestions on how we could approach this, or know of a similar example in the codebase that I could follow? I'm happy to rework the test in whatever direction you think is best. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
