yuqi1129 commented on code in PR #8777:
URL: https://github.com/apache/gravitino/pull/8777#discussion_r2497117760
##########
catalogs/hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/HDFSFileSystemProvider.java:
##########
@@ -28,6 +28,9 @@
import org.apache.hadoop.fs.Path;
public class HDFSFileSystemProvider implements FileSystemProvider {
+ public static final String IPC_FALLBACK_TO_SIMPLE_AUTH_ALLOWED =
+ "hadoop.rpc.protection.fallback-to-simple-auth-allowed";
Review Comment:
Is this the exact value that is defined in Hadoop?
##########
clients/filesystem-hadoop3/src/main/java/org/apache/gravitino/filesystem/hadoop/BaseGVFSOperations.java:
##########
@@ -729,6 +755,11 @@ protected FileSystem getActualFileSystemByLocationName(
Path targetLocation = new
Path(fileset.storageLocations().get(targetLocationName));
Map<String, String> allProperties =
getAllProperties(filesetIdent, targetLocation.toUri().getScheme(),
targetLocationName);
+ Schema schema =
getSchema(NameIdentifier.parse(filesetIdent.namespace().toString()));
+ allProperties.putAll(schema.properties());
Review Comment:
I wonder if the properties stored in the schema entity are in the format
like `fs.gravitino.fileset.properties.my_catalog.my_schema`?
##########
catalogs/hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/Constants.java:
##########
@@ -27,4 +27,25 @@ public class Constants {
// Name of the built-in HDFS file system provider
public static final String BUILTIN_HDFS_FS_PROVIDER = "builtin-hdfs";
+
+ // Name of the configuration property for HDFS config resources
+ public static final String HDFS_CONFIG_RESOURCES = "hdfs.config.resources";
+ // Name of the configuration property to disable HDFS FileSystem cache
+ public static final String FS_DISABLE_CACHE = "fs.hdfs.impl.disable.cache";
+ // Name of the configuration property for HDFS authentication type
+ public static final String HADOOP_SECURITY_AUTHENTICATION =
"hadoop.security.authentication";
+ // Name of the configuration property for Kerberos principal
+ public static final String HADOOP_SECURITY_PRINCIPAL =
+ "hadoop.security.authentication.kerberos.principal";
+ // Name of the configuration property for Kerberos keytab
+ public static final String HADOOP_SECURITY_KEYTAB =
+ "hadoop.security.authentication.kerberos.keytab";
+ // Name of the configuration property for Kerberos krb5.conf location
+ public static final String HADOOP_KRB5_CONF =
"hadoop.security.authentication.kerberos.krb5.conf";
+ // Environment variable for Java Kerberos configuration
+ public static final String SECURITY_KRB5_ENV = "java.security.krb5.conf";
+ // Supported authentication types
+ public static final String AUTH_KERBEROS = "kerberos";
Review Comment:
These two have been defined in `AuthenticationConfig`, could you reuse it?
##########
clients/filesystem-hadoop3/src/main/java/org/apache/gravitino/filesystem/hadoop/BaseGVFSOperations.java:
##########
@@ -661,6 +668,25 @@ protected Fileset getFileset(NameIdentifier filesetIdent) {
NameIdentifier.of(filesetIdent.namespace().level(2),
filesetIdent.name())));
}
+ /**
+ * Get the schema by the schema identifier from the cache or load it from
the server if the cache
Review Comment:
if the cache....? You may have missed the word after it.
##########
clients/filesystem-hadoop3/src/main/java/org/apache/gravitino/filesystem/hadoop/GravitinoVirtualFileSystemUtils.java:
##########
@@ -61,7 +61,16 @@ public static Map<String, String> getConfigMap(Configuration
configuration) {
Map<String, String> maps = Maps.newHashMap();
// Don't use entry.getKey() directly in the lambda, because it cannot
// handle variable expansion in the Configuration values.
- configuration.forEach(entry -> maps.put(entry.getKey(),
configuration.get(entry.getKey())));
+ Configuration defaultConf = new Configuration();
+ configuration.forEach(
Review Comment:
Since the logic and use scenario of this method have changed, you'd better
rename it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
