>From Hussain Towaileb <[email protected]>: Attention is currently required from: Murtadha Hubail, Michael Blow. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246 )
Change subject: [ASTERIXDB-3514][EXT]: Support trust auth for parquet + delete assumed creds on collection drop ...................................................................... Patch Set 7: (8 comments) File asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalCredentialsCache.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/8dbfab31_682c1678 PS7, Line 43: refreshAwsAssumeRolePercentage rename to refreshAwsAssumeRoleThreshold File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/ExternalProperties.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/1d6a90ce_d50005e9 PS7, Line 62: AWS_REFRESH_ASSUME_ROLE_PERCENTAGE rename to AWS_REFRESH_ASSUME_ROLE_THRESHOLD File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/aws/s3/S3AuthUtils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/e7b3195c_2a880102 PS7, Line 382: if (serviceEndpoint != null) Set the region property manually as well, we always have it https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/b2b49277_c7843f1d PS7, Line 421: jobConf.set(HADOOP_ASSUME_ROLE_SESSION_DURATION, "15m"); Range that hadoop supports is 15m to 1h (in this format), We should compare it to what's configured in the external property, if it is less than 1h, we will set it to match the configuration, however, if the configuration has more than 1 hour duration, we'll set the hadoop to 1h since this is the max. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/f02666d7_cdeee8e1 PS7, Line 423: // TODO: this assumes basic keys always, also support if we use InstanceProfile to assume a role We can use the following to assume a role, support them all: - Pass permanent credentials - Pass credentials in an environment variables instead - Use AWS instance profile File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/aws/s3/S3Constants.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/6b8f1778_21c4dcf6 PS7, Line 38: assumeRoleRefreshPercentage assumeRoleRefreshThreshold https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/7a93568a_b8377bd3 PS7, Line 50: AWS connectors for s3 and s3n are deprecated. Check if we're still using s3 or s3n anywhere, might have all been replaced to use s3a https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246/comment/bcdf2b72_83b7d8dc PS7, Line 59: public static final String HADOOP_REGION = "fs.s3a.region"; Currently unused, S3 probably uses bucket region detection to know the right location, set this if available. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19246 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I77998a5dfcc304692e12280b7b4018f3593085b9 Gerrit-Change-Number: 19246 Gerrit-PatchSet: 7 Gerrit-Owner: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Michael Blow <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Attention: Murtadha Hubail <[email protected]> Gerrit-Attention: Michael Blow <[email protected]> Gerrit-Comment-Date: Mon, 13 Jan 2025 15:43:03 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment
