Samuel Verstraete created SOLR-18610:
----------------------------------------
Summary: S3BackupRepository ignores web identity (IRSA)
credentials when the Security Manager is enabled, since the AWS SDK upgrade to
2.53
Key: SOLR-18610
URL: https://issues.apache.org/jira/browse/SOLR-18610
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: Backup/Restore
Affects Versions: 9.11
Reporter: Samuel Verstraete
With the Security Manager enabled (the default), the S3 backup repository in
Solr 9.11.0 no longer authenticates with web identity credentials
(\{{AWS_ROLE_ARN}} + \{{AWS_WEB_IDENTITY_TOKEN_FILE}}, as set by EKS IAM Roles
for Service Accounts). The web identity provider fails, the default credential
chain falls through to the next provider, and S3 requests are signed with the
EC2 instance profile instead. Backups, restores and LISTBACKUP then fail with a
403 for the node's role, or silently run as the node's role when that role
happens to have access to the bucket.
The same setup works on 9.10.1 and earlier.
h3. Steps to reproduce
Solr 9.11.0 (official image) in SolrCloud on EKS, with a pod service account
mapped to an IAM role that can read and write the backup bucket, and the node's
instance role without access to it. Default Security Manager,
\{{-Dsolr.modules=s3-repository}} and
\{{-Dsolr.allowPaths=/var/run/secrets/eks.amazonaws.com/serviceaccount/token}}.
An S3 repository in solr.xml with only \{{s3.bucket.name}} and \{{s3.region}}.
{code}
/solr/admin/collections?action=LISTBACKUP&name=<backup>&repository=s3&location=/
{code}
Returns a 500 with:
{code}
An AmazonServiceException was thrown! [serviceName=S3] [httpStatus=403]
[s3ErrorCode=AccessDenied] [message=User:
arn:aws:sts::<account>:assumed-role/<node instance role>/<instance id> is not
authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::<bucket>" ...]
{code}
The pod has \{{AWS_ROLE_ARN}} and \{{AWS_WEB_IDENTITY_TOKEN_FILE}} set, and
\{{sts-2.53.1.jar}} is in the module's lib directory.
The failure is easier to see outside Solr. A class that calls
\{{WebIdentityTokenFileCredentialsProvider.create().resolveCredentials()}}, run
in the same container with the module's jars on the classpath and the JVM
options Solr starts with (\{{-Djava.security.manager
-Djava.security.policy=server/etc/security.policy ...}}):
{code}
java.lang.IllegalStateException: Apache5HttpClient requires
jdk.net.NetworkPermission for "setOption.TCP_KEEPIDLE",
"setOption.TCP_KEEPINTERVAL", "setOption.TCP_KEEPCOUNT" when a SecurityManager
is active.
Caused by: java.security.AccessControlException: access denied
("jdk.net.NetworkPermission" "setOption.TCP_KEEPIDLE")
{code}
Without the Security Manager, the same class resolves the role from
\{{AWS_ROLE_ARN}}.
h3. Cause
Since AWS SDK 2.46, Apache HttpClient 5 (\{{apache5-client}}) is the default
sync HTTP client, and it comes in transitively, so the s3-repository module
ships it next to \{{apache-client}} (9.11.0: both 2.53.1). \{{S3StorageClient}}
passes an explicit \{{ApacheHttpClient}} (HttpClient 4) to the S3 client, so S3
calls themselves are unaffected. The web identity provider in the default
credential chain, however, builds its own STS client with the SDK's default
HTTP client, now Apache 5. HttpClient 5.6 sets TCP keep-alive socket options,
which need \{{jdk.net.NetworkPermission}} under a Security Manager.
\{{server/etc/security.policy}} does not grant them, so building the STS client
fails, and \{{DefaultCredentialsProvider}} moves on to the instance profile.
The SDK documents the required permissions in its announcement of the Apache 5
default: https://github.com/aws/aws-sdk-java-v2/issues/7007 (and fails fast on
them since https://github.com/aws/aws-sdk-java-v2/pull/6992).
The S3 repository tests do not catch this: they use static credentials against
S3Mock and never go through the web identity provider.
h3. Proposed fix
Grant the three permissions in \{{server/etc/security.policy}}, next to the
existing entries for the AWS SDK:
{code}
permission jdk.net.NetworkPermission "setOption.TCP_KEEPIDLE";
permission jdk.net.NetworkPermission "setOption.TCP_KEEPINTERVAL";
permission jdk.net.NetworkPermission "setOption.TCP_KEEPCOUNT";
{code}
Tested: with these three grants added to the policy, the web identity provider
gets past the permission check and calls STS.
An alternative is to exclude \{{software.amazon.awssdk:apache5-client}} from
the s3-repository dependencies, so the SDK's default HTTP client is HttpClient
4 again, which the module already depends on. The AWS SDK recommends moving to
Apache 5, so the policy change seems the better fix.
h3. Workaround
Either of these restores web identity credentials on 9.11.0, with the Security
Manager left on:
* Make the SDK use HttpClient 4 for every client it creates: add
\{{-Dsoftware.amazon.awssdk.http.service.impl=software.amazon.awssdk.http.apache.ApacheSdkHttpService}}
to \{{SOLR_OPTS}}. HttpClient 4 is in Solr's webapp lib. Verified on a 6-node
SolrCloud: LISTBACKUP, the nightly backups and a full collection RESTORE work
again with the pod's role.
* Grant the three permissions in an additional policy file. Solr adds its
policy with \{{-Djava.security.policy=}} (a single \{{=}}), so the JDK's
default \{{policy.url.2=file:${user.home}/.java.policy}} still applies; a file
at \{{/home/solr/.java.policy}} in the official image with only the three
grants is enough.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
