[
https://issues.apache.org/jira/browse/SOLR-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18610:
----------------------------------
Labels: pull-request-available (was: )
> S3BackupRepository ignores web identity (IRSA) credentials when the Security
> Manager is enabled, since the AWS SDK upgrade to 2.53
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18610
> URL: https://issues.apache.org/jira/browse/SOLR-18610
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
> Affects Versions: 9.11
> Reporter: Samuel Verstraete
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the Security Manager enabled (the default), the S3 backup repository in
> Solr 9.11.0 no longer authenticates with web identity credentials
> (\{{AWS_ROLE_ARN}} + \{{AWS_WEB_IDENTITY_TOKEN_FILE}}, as set by EKS IAM
> Roles for Service Accounts). The web identity provider fails, the default
> credential chain falls through to the next provider, and S3 requests are
> signed with the EC2 instance profile instead. Backups, restores and
> LISTBACKUP then fail with a 403 for the node's role, or silently run as the
> node's role when that role happens to have access to the bucket.
> The same setup works on 9.10.1 and earlier.
> h3. Steps to reproduce
> Solr 9.11.0 (official image) in SolrCloud on EKS, with a pod service account
> mapped to an IAM role that can read and write the backup bucket, and the
> node's instance role without access to it. Default Security Manager,
> \{{-Dsolr.modules=s3-repository}} and
> \{{-Dsolr.allowPaths=/var/run/secrets/eks.amazonaws.com/serviceaccount/token}}.
> An S3 repository in solr.xml with only \{{s3.bucket.name}} and
> \{{s3.region}}.
> {code}
> /solr/admin/collections?action=LISTBACKUP&name=<backup>&repository=s3&location=/
> {code}
> Returns a 500 with:
> {code}
> An AmazonServiceException was thrown! [serviceName=S3] [httpStatus=403]
> [s3ErrorCode=AccessDenied] [message=User:
> arn:aws:sts::<account>:assumed-role/<node instance role>/<instance id> is not
> authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::<bucket>" ...]
> {code}
> The pod has \{{AWS_ROLE_ARN}} and \{{AWS_WEB_IDENTITY_TOKEN_FILE}} set, and
> \{{sts-2.53.1.jar}} is in the module's lib directory.
> The failure is easier to see outside Solr. A class that calls
> \{{WebIdentityTokenFileCredentialsProvider.create().resolveCredentials()}},
> run in the same container with the module's jars on the classpath and the JVM
> options Solr starts with (\{{-Djava.security.manager
> -Djava.security.policy=server/etc/security.policy ...}}):
> {code}
> java.lang.IllegalStateException: Apache5HttpClient requires
> jdk.net.NetworkPermission for "setOption.TCP_KEEPIDLE",
> "setOption.TCP_KEEPINTERVAL", "setOption.TCP_KEEPCOUNT" when a
> SecurityManager is active.
> Caused by: java.security.AccessControlException: access denied
> ("jdk.net.NetworkPermission" "setOption.TCP_KEEPIDLE")
> {code}
> Without the Security Manager, the same class resolves the role from
> \{{AWS_ROLE_ARN}}.
> h3. Cause
> Since AWS SDK 2.46, Apache HttpClient 5 (\{{apache5-client}}) is the default
> sync HTTP client, and it comes in transitively, so the s3-repository module
> ships it next to \{{apache-client}} (9.11.0: both 2.53.1).
> \{{S3StorageClient}} passes an explicit \{{ApacheHttpClient}} (HttpClient 4)
> to the S3 client, so S3 calls themselves are unaffected. The web identity
> provider in the default credential chain, however, builds its own STS client
> with the SDK's default HTTP client, now Apache 5. HttpClient 5.6 sets TCP
> keep-alive socket options, which need \{{jdk.net.NetworkPermission}} under a
> Security Manager. \{{server/etc/security.policy}} does not grant them, so
> building the STS client fails, and \{{DefaultCredentialsProvider}} moves on
> to the instance profile.
> The SDK documents the required permissions in its announcement of the Apache
> 5 default: https://github.com/aws/aws-sdk-java-v2/issues/7007 (and fails fast
> on them since https://github.com/aws/aws-sdk-java-v2/pull/6992).
> The S3 repository tests do not catch this: they use static credentials
> against S3Mock and never go through the web identity provider.
> h3. Proposed fix
> Grant the three permissions in \{{server/etc/security.policy}}, next to the
> existing entries for the AWS SDK:
> {code}
> permission jdk.net.NetworkPermission "setOption.TCP_KEEPIDLE";
> permission jdk.net.NetworkPermission "setOption.TCP_KEEPINTERVAL";
> permission jdk.net.NetworkPermission "setOption.TCP_KEEPCOUNT";
> {code}
> Tested: with these three grants added to the policy, the web identity
> provider gets past the permission check and calls STS.
> An alternative is to exclude \{{software.amazon.awssdk:apache5-client}} from
> the s3-repository dependencies, so the SDK's default HTTP client is
> HttpClient 4 again, which the module already depends on. The AWS SDK
> recommends moving to Apache 5, so the policy change seems the better fix.
> h3. Workaround
> Either of these restores web identity credentials on 9.11.0, with the
> Security Manager left on:
> * Make the SDK use HttpClient 4 for every client it creates: add
> \{{-Dsoftware.amazon.awssdk.http.service.impl=software.amazon.awssdk.http.apache.ApacheSdkHttpService}}
> to \{{SOLR_OPTS}}. HttpClient 4 is in Solr's webapp lib. Verified on a
> 6-node SolrCloud: LISTBACKUP, the nightly backups and a full collection
> RESTORE work again with the pod's role.
> * Grant the three permissions in an additional policy file. Solr adds its
> policy with \{{-Djava.security.policy=}} (a single \{{=}}), so the JDK's
> default \{{policy.url.2=file:${user.home}/.java.policy}} still applies; a
> file at \{{/home/solr/.java.policy}} in the official image with only the
> three grants is enough.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
