[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759187#comment-17759187 ] ASF GitHub Bot commented on HDFS-17148: --- goiri merged PR #5966: URL: https://github.com/apache/hadoop/pull/5966 > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756581#comment-17756581 ] ASF GitHub Bot commented on HDFS-17148: --- hchaverri commented on PR #5966: URL: https://github.com/apache/hadoop/pull/5966#issuecomment-1685390520 @goiri Could you help merge this PR into branch-3.3? > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756444#comment-17756444
]
ASF GitHub Bot commented on HDFS-17148:
---
hadoop-yetus commented on PR #5966:
URL: https://github.com/apache/hadoop/pull/5966#issuecomment-1685180147
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 47s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 51s | | branch-3.3 passed |
| +1 :green_heart: | compile | 12m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 1m 44s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 1m 56s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 1m 43s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 2m 47s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 24m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 11m 56s | | the patch passed |
| +1 :green_heart: | javac | 11m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 36s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 14m 53s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 167m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5966/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5966 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d6494eb9fb98 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 34faeee5cf8312a56719c87bba8cfffaf652ee52 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5966/1/testReport/ |
| Max. process+thread count | 2413 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5966/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Assignee: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
>
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756432#comment-17756432 ] ASF GitHub Bot commented on HDFS-17148: --- hchaverri opened a new pull request, #5966: URL: https://github.com/apache/hadoop/pull/5966 ### Description of PR This is a backport to branch-3.3 of commit: https://github.com/apache/hadoop/commit/ad2f45c64f01a520a01f6876d8493140b9f89b03 Patch applied cleanly except for checkstyle warnings on test class (addressed on commit 34faeee) JIRA: [HDFS-17148](https://issues.apache.org/jira/browse/HDFS-17148). RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL These changes update the SQLDelegationTokenSecretManager to cleanup expired tokens found in SQL. Currently, AbstractDelegationTokenSecretManagers only cleanup tokens in its memory cache. The SQLDelegationTokenSecretManager was recently updated to use a LoadingCache with a short TTL, so most expired tokens won't be present in memory. During token cleanup, the SQLDelegationTokenSecretManager will query SQL for a list of tokens that have not been updated recently, based on the modifiedTime column. We will limit the amount of results returned to prevent performance impact on SQL. Once the list is returned, the ExpiredTokenRemover will evaluate if the tokens are actually expired and delete them from SQL if so. ### How was this patch tested? Added unit test for different token cleanup scenarios: Having an expired token in SQL. which should be deleted Having a token with a long renewal time, which should not be deleted Having a token recently renewed, which should not be deleted ### For code changes: - [Y] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [Y] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [Y] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [Y] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753338#comment-17753338 ] ASF GitHub Bot commented on HDFS-17148: --- goiri merged PR #5936: URL: https://github.com/apache/hadoop/pull/5936 > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753003#comment-17753003
]
ASF GitHub Bot commented on HDFS-17148:
---
hadoop-yetus commented on PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#issuecomment-1674107480
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 38m 29s | | trunk passed |
| +1 :green_heart: | compile | 19m 43s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 5s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 43m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 41s | | the patch passed |
| +1 :green_heart: | compile | 18m 47s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 18m 47s | | the patch passed |
| +1 :green_heart: | compile | 17m 14s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 17m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 3s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 49s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 22m 50s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 294m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5585ce4906fe 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a7986ed4b96b7502709c6158007b764b22adfcc7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/4/testReport/ |
| Max. process+thread count | 2918 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/4/console |
|
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752996#comment-17752996
]
ASF GitHub Bot commented on HDFS-17148:
---
hadoop-yetus commented on PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#issuecomment-1674079972
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 54s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 42s | | trunk passed |
| +1 :green_heart: | compile | 19m 12s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 18m 13s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 40s | | the patch passed |
| +1 :green_heart: | compile | 19m 10s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 19m 10s | | the patch passed |
| +1 :green_heart: | compile | 17m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 17m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 22m 34s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 292m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fc9f91861e77 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ccb908b3957b707a686739f3bbe8caf8d6bf49f4 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/3/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/3/console |
|
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752925#comment-17752925
]
ASF GitHub Bot commented on HDFS-17148:
---
hchaverri commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290725889
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -458,8 +458,8 @@ public int incrementCurrentKeyId() {
// Token operations in SQL database
protected abstract byte[] selectTokenInfo(int sequenceNum, byte[]
tokenIdentifier)
throws SQLException;
- protected abstract Map selectTokenInfos(long
maxModifiedTime, int topResults)
- throws SQLException;
+ protected abstract Map selectStaleTokenInfos(long
maxModifiedTime,
+ int topResults) throws SQLException;
Review Comment:
Makes sense, I've renamed it
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752915#comment-17752915
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290614284
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -458,8 +458,8 @@ public int incrementCurrentKeyId() {
// Token operations in SQL database
protected abstract byte[] selectTokenInfo(int sequenceNum, byte[]
tokenIdentifier)
throws SQLException;
- protected abstract Map selectTokenInfos(long
maxModifiedTime, int topResults)
- throws SQLException;
+ protected abstract Map selectStaleTokenInfos(long
maxModifiedTime,
+ int topResults) throws SQLException;
Review Comment:
Nit: `topResults` implies ordering. Maybe use `maxResults` like you did for
the implementation.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752913#comment-17752913
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290611209
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java:
##
@@ -190,6 +190,14 @@ public long getCurrentTokensSize() {
return currentTokens.size();
}
+ /**
+ * Interval for tokens to be renewed.
+ * @return Renew interval in milliseconds.
+ */
+ protected long getTokenRenewInterval() {
Review Comment:
Yeah sure.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752911#comment-17752911
]
ASF GitHub Bot commented on HDFS-17148:
---
hchaverri commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290584078
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java:
##
@@ -190,6 +190,14 @@ public long getCurrentTokensSize() {
return currentTokens.size();
}
+ /**
+ * Interval for tokens to be renewed.
+ * @return Renew interval in milliseconds.
+ */
+ protected long getTokenRenewInterval() {
Review Comment:
Let's skip this if possible as there's a few other properties with the same
naming format
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752909#comment-17752909
]
ASF GitHub Bot commented on HDFS-17148:
---
hchaverri commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290584223
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java:
##
@@ -766,6 +774,10 @@ private void removeExpiredToken() throws IOException {
logExpireTokens(expiredTokens);
}
+ protected Map getTokensForCleanup() {
Review Comment:
Makes sense, I've renamed it
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -415,6 +458,8 @@ public int incrementCurrentKeyId() {
// Token operations in SQL database
protected abstract byte[] selectTokenInfo(int sequenceNum, byte[]
tokenIdentifier)
throws SQLException;
+ protected abstract Map selectTokenInfos(long
maxModifiedTime, int topResults)
Review Comment:
I thought that having the maxModifiedTime in the parameter list would make
it clear enough, but I'll still rename the method to prevent confusion
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -153,6 +163,39 @@ public synchronized TokenIdent
cancelToken(Token token,
return super.cancelToken(token, canceller);
}
+ /**
+ * Obtain a list of tokens that will be considered for cleanup, based on the
last
+ * time the token was updated in SQL. This list may include tokens that are
not
+ * expired and should not be deleted (e.g. if the token was last renewed
using a
+ * higher renewal interval).
+ * The number of results is limited to reduce performance impact. Some level
of
+ * contention is expected when multiple routers run cleanup simultaneously.
+ * @return Map of tokens that have not been updated in SQL after the token
renewal
+ * period.
+ */
+ @Override
+ protected Map getTokensForCleanup() {
+Map tokens = new HashMap<>();
+try {
+ // Query SQL for tokens that haven't been updated after
+ // the last token renewal period.
+ long maxModifiedTime = Time.now() - getTokenRenewInterval();
+ Map tokenInfoBytesList =
selectTokenInfos(maxModifiedTime,
+ this.maxTokenCleanupResults);
+
+ LOG.info("Found {} tokens for cleanup", tokenInfoBytesList.size());
+ for (Map.Entry tokenInfoBytes :
tokenInfoBytesList.entrySet()) {
+TokenIdent tokenIdent = createTokenIdent(tokenInfoBytes.getKey());
+DelegationTokenInformation tokenInfo =
createTokenInfo(tokenInfoBytes.getValue());
+tokens.put(tokenIdent, tokenInfo);
+ }
+} catch (IOException | SQLException e) {
+ LOG.error("Failed to get all tokens in SQL secret manager", e);
Review Comment:
Good catch. I'll change this
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752900#comment-17752900
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290503777
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -415,6 +458,8 @@ public int incrementCurrentKeyId() {
// Token operations in SQL database
protected abstract byte[] selectTokenInfo(int sequenceNum, byte[]
tokenIdentifier)
throws SQLException;
+ protected abstract Map selectTokenInfos(long
maxModifiedTime, int topResults)
Review Comment:
Consider renaming to `selectUnrenewedTokenInfos` of something that indicates
the subset of tokens we are fetching.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752898#comment-17752898
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1289577317
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java:
##
@@ -190,6 +190,14 @@ public long getCurrentTokensSize() {
return currentTokens.size();
}
+ /**
+ * Interval for tokens to be renewed.
+ * @return Renew interval in milliseconds.
+ */
+ protected long getTokenRenewInterval() {
Review Comment:
You can add a unit on `tokenRenewInterval` too.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752899#comment-17752899
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290503777
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -415,6 +458,8 @@ public int incrementCurrentKeyId() {
// Token operations in SQL database
protected abstract byte[] selectTokenInfo(int sequenceNum, byte[]
tokenIdentifier)
throws SQLException;
+ protected abstract Map selectTokenInfos(long
maxModifiedTime, int topResults)
Review Comment:
Consider renaming to `selectUnrenewedTokenInfos` or something that indicates
the subset of tokens we are fetching.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752897#comment-17752897
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290501755
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##
@@ -153,6 +163,39 @@ public synchronized TokenIdent
cancelToken(Token token,
return super.cancelToken(token, canceller);
}
+ /**
+ * Obtain a list of tokens that will be considered for cleanup, based on the
last
+ * time the token was updated in SQL. This list may include tokens that are
not
+ * expired and should not be deleted (e.g. if the token was last renewed
using a
+ * higher renewal interval).
+ * The number of results is limited to reduce performance impact. Some level
of
+ * contention is expected when multiple routers run cleanup simultaneously.
+ * @return Map of tokens that have not been updated in SQL after the token
renewal
+ * period.
+ */
+ @Override
+ protected Map getTokensForCleanup() {
+Map tokens = new HashMap<>();
+try {
+ // Query SQL for tokens that haven't been updated after
+ // the last token renewal period.
+ long maxModifiedTime = Time.now() - getTokenRenewInterval();
+ Map tokenInfoBytesList =
selectTokenInfos(maxModifiedTime,
+ this.maxTokenCleanupResults);
+
+ LOG.info("Found {} tokens for cleanup", tokenInfoBytesList.size());
+ for (Map.Entry tokenInfoBytes :
tokenInfoBytesList.entrySet()) {
+TokenIdent tokenIdent = createTokenIdent(tokenInfoBytes.getKey());
+DelegationTokenInformation tokenInfo =
createTokenInfo(tokenInfoBytes.getValue());
+tokens.put(tokenIdent, tokenInfo);
+ }
+} catch (IOException | SQLException e) {
+ LOG.error("Failed to get all tokens in SQL secret manager", e);
Review Comment:
This is not `all tokens` but a subset, filtered my maxModifiedTime.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752896#comment-17752896
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1289577122
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java:
##
@@ -190,6 +190,14 @@ public long getCurrentTokensSize() {
return currentTokens.size();
}
+ /**
+ * Interval for tokens to be renewed.
+ * @return Renew interval in milliseconds.
+ */
+ protected long getTokenRenewInterval() {
Review Comment:
Please add the unit in method name `getTokenRenewalIntervalMs`
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752895#comment-17752895
]
ASF GitHub Bot commented on HDFS-17148:
---
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290498330
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java:
##
@@ -766,6 +774,10 @@ private void removeExpiredToken() throws IOException {
logExpireTokens(expiredTokens);
}
+ protected Map getTokensForCleanup() {
Review Comment:
Maybe rename to `getCandidateTokensForCleanup()` so it is clear there are
other checks that will happen.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752625#comment-17752625 ] ASF GitHub Bot commented on HDFS-17148: --- slfan1989 commented on PR #5936: URL: https://github.com/apache/hadoop/pull/5936#issuecomment-1672563962 @goiri @simbadzina Can you help to review this pr? Thank you very much! From my personal point of view, this PR is good. > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752622#comment-17752622
]
ASF GitHub Bot commented on HDFS-17148:
---
hadoop-yetus commented on PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#issuecomment-167274
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 30s | | trunk passed |
| +1 :green_heart: | compile | 18m 29s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 59s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 17m 33s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 17m 33s | | the patch passed |
| +1 :green_heart: | compile | 16m 59s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 59s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 53s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 22m 20s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 281m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e66ca09926f6 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0c721a2df50771f821585ab1a47657081aaac84d |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/2/testReport/ |
| Max. process+thread count | 2652 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/2/console |
|
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752539#comment-17752539
]
ASF GitHub Bot commented on HDFS-17148:
---
hadoop-yetus commented on PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#issuecomment-1672270862
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 3s | | trunk passed |
| +1 :green_heart: | compile | 19m 16s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 18s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 59s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 17m 39s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 17m 39s | | the patch passed |
| +1 :green_heart: | compile | 17m 14s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 17m 14s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 4m 33s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 1m 44s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) |
| +1 :green_heart: | shadedclient | 39m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 53s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 22m 24s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 299m 5s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| |
org.apache.hadoop.hdfs.server.federation.router.security.token.SQLDelegationTokenSecretManagerImpl.lambda$selectTokenInfos$4(long,
int) may fail to clean up java.sql.ResultSet Obligation to clean up resource
created at SQLDelegationTokenSecretManagerImpl.java:clean up java.sql.ResultSet
Obligation to clean up resource created at
SQLDelegationTokenSecretManagerImpl.java:[line 165] is not discharged |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5936/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5936 |
| Optional Tests | dupname asflicense compile javac
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752534#comment-17752534 ] ASF GitHub Bot commented on HDFS-17148: --- slfan1989 commented on PR #5936: URL: https://github.com/apache/hadoop/pull/5936#issuecomment-1672259214 Good catch! > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752489#comment-17752489 ] ASF GitHub Bot commented on HDFS-17148: --- hchaverri opened a new pull request, #5936: URL: https://github.com/apache/hadoop/pull/5936 ### Description of PR JIRA: HDFS-17148. RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL These changes update the SQLDelegationTokenSecretManager to cleanup expired tokens found in SQL. Currently, AbstractDelegationTokenSecretManagers only cleanup tokens in its memory cache. The SQLDelegationTokenSecretManager was recently updated to use a LoadingCache with a short TTL, so most expired tokens won't be present in memory. During token cleanup, the SQLDelegationTokenSecretManager will query SQL for a list of tokens that have not been updated recently, based on the modifiedTime column. We will limit the amount of results returned to prevent performance impact on SQL. Once the list is returned, the ExpiredTokenRemover will evaluate if the tokens are actually expired and delete them from SQL if so. ### How was this patch tested? Added unit test for different token cleanup scenarios: 1. Having an expired token in SQL. which should be deleted 2. Having a token with a long renewal time, which should not be deleted 3. Having a token recently renewed, which should not be deleted ### For code changes: - [Y] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [Y] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [Y] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [Y] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Priority: Major > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
