[jira] [Commented] (HDFS-17529) Improve router state store cache entry deletion
[
https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848047#comment-17848047
]
ASF GitHub Bot commented on HDFS-17529:
---
hadoop-yetus commented on PR #6833:
URL: https://github.com/apache/hadoop/pull/6833#issuecomment-2121778835
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| -1 :x: | javadoc | 0m 28s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 33m 38s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 177m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
markdownlint |
| uname | Linux 1ecfff136614 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 00bca37b88cdf179a429030c4b53fc2c69e2ef54 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/3/testReport/
[jira] [Commented] (HDFS-17529) Improve router state store cache entry deletion
[ https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848038#comment-17848038 ] ASF GitHub Bot commented on HDFS-17529: --- ZanderXu commented on code in PR #6833: URL: https://github.com/apache/hadoop/pull/6833#discussion_r1607542930 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/StateStoreRecordOperations.java: ## @@ -127,6 +128,17 @@ StateStoreOperationResult putAll( @AtMostOnce boolean remove(T record) throws IOException; + /** + * Remove multiple records. + * + * @param Record class of the records. + * @param records Records to be removed. + * @return Map of record -> boolean indicating any entries being deleted by this record. Review Comment: `Map of record -> boolean indicating if the record has being removed successfully` ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/StateStoreRecordOperations.java: ## @@ -152,4 +164,17 @@ StateStoreOperationResult putAll( int remove(Class clazz, Query query) throws IOException; + /** + * Remove all records of a specific class that match any query in a list of queries. + * Requires the getAll implementation to fetch fresh records on each call. + * + * @param clazz The class to match the records with. + * @param queries Queries (logical OR) to filter what to remove. + * @param Record class of the records. + * @return Map of query to number of records deleted by that query. Review Comment: `Map of query to number of records removed by that query.` > Improve router state store cache entry deletion > --- > > Key: HDFS-17529 > URL: https://issues.apache.org/jira/browse/HDFS-17529 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Major > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket aims to improve the deletion process for ZK state store > implementation. > See HDFS-17532 for the other half of this improvement -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17529) Improve router state store cache entry deletion
[
https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848029#comment-17848029
]
ASF GitHub Bot commented on HDFS-17529:
---
ZanderXu commented on code in PR #6833:
URL: https://github.com/apache/hadoop/pull/6833#discussion_r1607499422
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreBaseImpl.java:
##
@@ -86,4 +89,37 @@ public boolean remove(T record)
throws IOException {
Class recordClass = (Class)StateStoreUtils.getRecordClass(clazz);
return remove(recordClass, query) == 1;
}
+
+ @Override
+ public Map removeMultiple(List
records) throws IOException {
+assert !records.isEmpty();
+// Fall back to iterative remove() calls if all records don't share 1 class
+Class expectedClazz = records.get(0).getClass();
+if (!records.stream().allMatch(x -> x.getClass() == expectedClazz)) {
+ Map result = new HashMap<>();
+ for (T record : records) {
+result.put(record, remove(record));
+ }
+ return result;
+}
+
+final List> queries = new ArrayList<>();
+for (T record: records) {
+ queries.add(new Query<>(record));
+}
+@SuppressWarnings("unchecked")
+Class recordClass = (Class)
StateStoreUtils.getRecordClass(expectedClazz);
+Map, Integer> result = remove(recordClass, queries);
+return result.entrySet().stream()
+.collect(Collectors.toMap(e -> e.getKey().getPartial(), e ->
e.getValue() > 0));
Review Comment:
`remove(T record)` returns true if `remove(recordClass, query)` is 1. But
here is `e.getValue() > 0`. So how about make them consistent?
Here, how about using `e.getValue() == 1`?
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/store/TestStateStoreMembershipState.java:
##
@@ -565,7 +568,7 @@ public void testRegistrationExpiredRaceCondition()
// Load cache
MembershipStore memStoreSpy = spy(membershipStore);
DelayAnswer delayer = new DelayAnswer(LOG);
-doAnswer(delayer).when(memStoreSpy).overrideExpiredRecords(any());
+doAnswer(delayer).when(memStoreSpy).overrideExpiredRecords(any(),
anyBoolean());
Review Comment:
remove this `anyBoolean()`
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -284,51 +288,88 @@ public StateStoreOperationResult
putAll(
}
@Override
- public int remove(
- Class clazz, Query query) throws IOException {
+ public Map, Integer> remove(Class clazz,
+ List> queries) throws IOException {
verifyDriverReady();
-if (query == null) {
- return 0;
+// Track how many entries are deleted by each query
+Map, Integer> ret = new HashMap<>();
+final List trueRemoved = Collections.synchronizedList(new
ArrayList<>());
+if (queries.isEmpty()) {
+ return ret;
}
// Read the current data
long start = monotonicNow();
-List records = null;
+List records;
try {
QueryResult result = get(clazz);
records = result.getRecords();
} catch (IOException ex) {
LOG.error("Cannot get existing records", ex);
getMetrics().addFailure(monotonicNow() - start);
- return 0;
+ return ret;
}
// Check the records to remove
String znode = getZNodeForClass(clazz);
-List recordsToRemove = filterMultiple(query, records);
+Set recordsToRemove = new HashSet<>();
+Map, List> queryToRecords = new HashMap<>();
+for (Query query : queries) {
+ List filtered = filterMultiple(query, records);
+ queryToRecords.put(query, filtered);
+ recordsToRemove.addAll(filtered);
+}
// Remove the records
-int removed = 0;
-for (T existingRecord : recordsToRemove) {
+List> callables = new ArrayList<>();
+recordsToRemove.forEach(existingRecord -> callables.add(() -> {
LOG.info("Removing \"{}\"", existingRecord);
try {
String primaryKey = getPrimaryKey(existingRecord);
String path = getNodePath(znode, primaryKey);
if (zkManager.delete(path)) {
- removed++;
+ trueRemoved.add(existingRecord);
} else {
LOG.error("Did not remove \"{}\"", existingRecord);
}
} catch (Exception e) {
LOG.error("Cannot remove \"{}\"", existingRecord, e);
getMetrics().addFailure(monotonicNow() - start);
}
+ return null;
+}));
+try {
+ if (enableConcurrent) {
+executorService.invokeAll(callables);
+ } else {
+for (Callable callable : callables) {
+ callable.call();
+}
+ }
+} catch (Exception e) {
+ LOG.error("Record removal failed : {}", e.getMessage(), e);
+ throw new
[jira] [Updated] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-17464: -- Target Version/s: 3.5.0 > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.5.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-17464: -- Affects Version/s: 3.5.0 (was: 3.4.0) > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.5.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu resolved HDFS-17464. --- Fix Version/s: 3.5.0 Resolution: Resolved > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848024#comment-17848024 ] ASF GitHub Bot commented on HDFS-17464: --- haiyang1987 commented on PR #6724: URL: https://github.com/apache/hadoop/pull/6724#issuecomment-2121550133 Committed to trunk. Thanks @hfutatzhanghb for your contributions and @ZanderXu @ayushtkn review! > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848023#comment-17848023 ] ASF GitHub Bot commented on HDFS-17464: --- haiyang1987 merged PR #6724: URL: https://github.com/apache/hadoop/pull/6724 > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17531) RBF: Asynchronous router RPC.
[
https://issues.apache.org/jira/browse/HDFS-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848017#comment-17848017
]
ASF GitHub Bot commented on HDFS-17531:
---
slfan1989 commented on PR #6838:
URL: https://github.com/apache/hadoop/pull/6838#issuecomment-2121500478
> @ayushtkn @slfan1989 hi, thanks for you replay, I sent the discussion to
[common-...@hadoop.apache.org](mailto:common-...@hadoop.apache.org).
This pr has too many changes and affects multiple modules, causing the
compilation to time out. I have seen the discussion emails, and the usual
discussion process may take 5-7 days.
> RBF: Asynchronous router RPC.
> -
>
> Key: HDFS-17531
> URL: https://issues.apache.org/jira/browse/HDFS-17531
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Aynchronous router.pdf, HDFS-17531.001.patch,
> image-2024-05-19-18-07-51-282.png
>
>
> *Description*
> Currently, the main function of the Router service is to accept client
> requests, forward the requests to the corresponding downstream ns, and then
> return the results of the downstream ns to the client. The link is as follows:
> *!image-2024-05-19-18-07-51-282.png|width=900,height=300!*
> The main threads involved in the rpc link are:
> {*}Read{*}: Get the client request and put it into the call queue *(1)*
> {*}Handler{*}:
> Extract call *(2)* from the call queue, process the call, generate a new
> call, place it in the call of the connection thread, and wait for the call
> processing to complete *(3)*
> After being awakened by the connection thread, process the response and put
> it into the response queue *(5)*
> *Connection:*
> Hold the link with downstream ns, send the call from the call to the
> downstream ns (via {*}rpcRequestThread{*}), and obtain a response from ns.
> Based on the call in the response, notify the call to complete processing
> *(4)*
> *Responder:*
> Retrieve the response queue from the queue *(6)* and return it to the client
>
> *Shortcoming*
> Even if the *connection* thread can send more requests to downstream
> nameservices, since *(3)* and *(4)* are synchronous, when the *handler*
> thread adds the call to connection.calls, it needs to wait until the
> *connection* notifies the call to complete, and then Only after the response
> is put into the response queue can a new call be obtained from the call queue
> and processed. Therefore, the concurrency performance of the router is
> limited by the number of handlers; a simple example is as follows: If the
> number of handlers is 1 and the maximum number of calls in the connection
> thread is 10, then even if the connection thread can send 10 requests to the
> downstream ns, since the number of handlers is 1, the router can only process
> one request after another.
>
> Since the performance of router rpc is mainly limited by the number of
> handlers, the most effective way to improve rpc performance currently is to
> increase the number of handlers. Letting the router create a large number of
> handler threads will also increase the number of thread switches and cannot
> maximize the use of machine performance.
>
> There are usually multiple ns downstream of the router. If the handler
> forwards the request to an ns with poor performance, it will cause the
> handler to wait for a long time. Due to the reduction of available handlers,
> the router's ability to handle ns requests with normal performance will be
> reduced. From the perspective of the client, the performance of the
> downstream ns of the router has deteriorated at this time. We often find that
> the call queue of the downstream ns is not high, but the call queue of the
> router is very high.
>
> Therefore, although the main function of the router is to federate and handle
> requests from multiple NSs, the current synchronous RPC performance cannot
> satisfy the scenario where there are many NSs downstream of the router. Even
> if the concurrent performance of the router can be improved by increasing the
> number of handlers, it is still relatively slow. More threads will increase
> the CPU context switching time, and in fact many of the handler threads are
> in a blocked state, which is undoubtedly a waste of thread resources. When a
> request enters the router, there is no guarantee that there will be a running
> handler at this time.
>
> Therefore, I consider asynchronous router rpc. Please view the *pdf* for the
> complete solution.
>
> Welcome everyone to exchange and discuss!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe,
[jira] [Commented] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848013#comment-17848013
]
Shilun Fan commented on HDFS-17533:
---
[~simbadzina] Thank you for the feedback! The upgrade from derby 10.14.2.0 to
10.17.1.0 was completed by us at [https://github.com/apache/hadoop/pull/6816],
and no abnormal unit tests were found at that time. I will roll back #6816.
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848003#comment-17848003
]
Simbarashe Dzinamarira edited comment on HDFS-17533 at 5/20/24 11:44 PM:
-
W.r.t to the solution, EmbeddedDriver was moved to the jerbytools jar so we
need to declare that dependency in the pom.xml.
When I include derbytools, I get the following error.
{noformat}
[ERROR]
/Users/sdzinama/dev/hadooptree/simbatrunk/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreMySQL.java:[42,30]
cannot access org.apache.derby.jdbc.EmbeddedDriver
[ERROR] bad class file:
/Users/sdzinama/.m2/repository/org/apache/derby/derbytools/10.17.1.0/derbytools-10.17.1.0.jar(org/apache/derby/jdbc/EmbeddedDriver.class)
[ERROR] class file has wrong version 63.0, should be 52.0
[ERROR] Please remove or make sure it appears in the correct subdirectory
of the classpath.{noformat}
was (Author: simbadzina):
W.r.t to the solution, EmbeddedDriver was moved to the jerbytools jar so we
need to declare that dependency in the pom.xml
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848006#comment-17848006
]
Simbarashe Dzinamarira edited comment on HDFS-17533 at 5/20/24 11:43 PM:
-
[~slfan1989] the following PR updated derby from 10.14.2.0 to 10.17.1.0.
[https://github.com/apache/hadoop/pull/6816]
However 10.17.1.0 requires a higher java version.
Any recommendation on how to resolve this? I assume downgrading is not an
option.
was (Author: simbadzina):
[~slfan1989] the following PR updated derby from 10.14.2.0 to 10.17.1.0.
[https://github.com/apache/hadoop/pull/6816]
However 10.17.1.0 requires a higher java version.
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848006#comment-17848006
]
Simbarashe Dzinamarira commented on HDFS-17533:
---
[~slfan1989] the following PR updated derby from 10.14.2.0 to 10.17.1.0.
[https://github.com/apache/hadoop/pull/6816]
However 10.17.1.0 requires a higher java version.
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848003#comment-17848003
]
Simbarashe Dzinamarira commented on HDFS-17533:
---
W.r.t to the solution, EmbeddedDriver was moved to the jerbytools jar so we
need to declare that dependency in the pom.xml
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simbarashe Dzinamarira updated HDFS-17533:
--
Description:
In the CI runs for RBF the following two tests are failing
{noformat}
[ERROR] Failures:
[ERROR]
org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
[ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[INFO]
[ERROR]
org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
[ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true
[ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true
[ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true {noformat}
[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
I believe the fix is first registering the driver:
[https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
[https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
was:
In the CI runs for RBF the following two tests are failing
{noformat}
[ERROR] Failures:
[ERROR]
org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
[ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[INFO]
[ERROR]
org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
[ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true
[ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true
[ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true {noformat}
[jira] [Assigned] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simbarashe Dzinamarira reassigned HDFS-17533:
-
Assignee: Simbarashe Dzinamarira
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
>
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17533) RBF: Unit tests that use embedded SQL failing in CI
[
https://issues.apache.org/jira/browse/HDFS-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simbarashe Dzinamarira updated HDFS-17533:
--
Summary: RBF: Unit tests that use embedded SQL failing in CI (was: RBF
Tests that use embedded SQL failing unit tests)
> RBF: Unit tests that use embedded SQL failing in CI
> ---
>
> Key: HDFS-17533
> URL: https://issues.apache.org/jira/browse/HDFS-17533
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Simbarashe Dzinamarira
>Priority: Major
>
> In the CI runs for RBF the following two tests are failing
> {noformat}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
> [ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
> failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:TokenStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:TokenStore;drop=true
> [INFO]
> [ERROR]
> org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
> [ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true
> [ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
> java.sql.SQLException: No suitable driver found for
> jdbc:derby:memory:StateStore;create=true
> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
> found for jdbc:derby:memory:StateStore;drop=true {noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
>
> I believe the fix is first registering the driver:
> [https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
>
> [https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17533) RBF Tests that use embedded SQL failing unit tests
Simbarashe Dzinamarira created HDFS-17533:
-
Summary: RBF Tests that use embedded SQL failing unit tests
Key: HDFS-17533
URL: https://issues.apache.org/jira/browse/HDFS-17533
Project: Hadoop HDFS
Issue Type: Test
Reporter: Simbarashe Dzinamarira
In the CI runs for RBF the following two tests are failing
{noformat}
[ERROR] Failures:
[ERROR]
org.apache.hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl.null
[ERROR] Run 1: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[ERROR] Run 2: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[ERROR] Run 3: TestSQLDelegationTokenSecretManagerImpl Multiple Failures (2
failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:TokenStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:TokenStore;drop=true
[INFO]
[ERROR]
org.apache.hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL.null
[ERROR] Run 1: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true
[ERROR] Run 2: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true
[ERROR] Run 3: TestStateStoreMySQL Multiple Failures (2 failures)
java.sql.SQLException: No suitable driver found for
jdbc:derby:memory:StateStore;create=true
java.lang.RuntimeException: java.sql.SQLException: No suitable driver
found for jdbc:derby:memory:StateStore;drop=true {noformat}
[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6804/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt]
I believe the fix is first registering the driver:
[https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html]
[https://stackoverflow.com/questions/22384710/java-sql-sqlexception-no-suitable-driver-found-for-jdbcmysql-localhost3306]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847961#comment-17847961 ] ASF GitHub Bot commented on HDFS-17464: --- ayushtkn commented on PR #6724: URL: https://github.com/apache/hadoop/pull/6724#issuecomment-2121001399 @haiyang1987 / @ZanderXu anyone hitting the merge button? > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17531) RBF: Asynchronous router RPC.
[
https://issues.apache.org/jira/browse/HDFS-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simbarashe Dzinamarira updated HDFS-17531:
--
Summary: RBF: Asynchronous router RPC. (was: RBF: Aynchronous router RPC.)
> RBF: Asynchronous router RPC.
> -
>
> Key: HDFS-17531
> URL: https://issues.apache.org/jira/browse/HDFS-17531
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Aynchronous router.pdf, HDFS-17531.001.patch,
> image-2024-05-19-18-07-51-282.png
>
>
> *Description*
> Currently, the main function of the Router service is to accept client
> requests, forward the requests to the corresponding downstream ns, and then
> return the results of the downstream ns to the client. The link is as follows:
> *!image-2024-05-19-18-07-51-282.png|width=900,height=300!*
> The main threads involved in the rpc link are:
> {*}Read{*}: Get the client request and put it into the call queue *(1)*
> {*}Handler{*}:
> Extract call *(2)* from the call queue, process the call, generate a new
> call, place it in the call of the connection thread, and wait for the call
> processing to complete *(3)*
> After being awakened by the connection thread, process the response and put
> it into the response queue *(5)*
> *Connection:*
> Hold the link with downstream ns, send the call from the call to the
> downstream ns (via {*}rpcRequestThread{*}), and obtain a response from ns.
> Based on the call in the response, notify the call to complete processing
> *(4)*
> *Responder:*
> Retrieve the response queue from the queue *(6)* and return it to the client
>
> *Shortcoming*
> Even if the *connection* thread can send more requests to downstream
> nameservices, since *(3)* and *(4)* are synchronous, when the *handler*
> thread adds the call to connection.calls, it needs to wait until the
> *connection* notifies the call to complete, and then Only after the response
> is put into the response queue can a new call be obtained from the call queue
> and processed. Therefore, the concurrency performance of the router is
> limited by the number of handlers; a simple example is as follows: If the
> number of handlers is 1 and the maximum number of calls in the connection
> thread is 10, then even if the connection thread can send 10 requests to the
> downstream ns, since the number of handlers is 1, the router can only process
> one request after another.
>
> Since the performance of router rpc is mainly limited by the number of
> handlers, the most effective way to improve rpc performance currently is to
> increase the number of handlers. Letting the router create a large number of
> handler threads will also increase the number of thread switches and cannot
> maximize the use of machine performance.
>
> There are usually multiple ns downstream of the router. If the handler
> forwards the request to an ns with poor performance, it will cause the
> handler to wait for a long time. Due to the reduction of available handlers,
> the router's ability to handle ns requests with normal performance will be
> reduced. From the perspective of the client, the performance of the
> downstream ns of the router has deteriorated at this time. We often find that
> the call queue of the downstream ns is not high, but the call queue of the
> router is very high.
>
> Therefore, although the main function of the router is to federate and handle
> requests from multiple NSs, the current synchronous RPC performance cannot
> satisfy the scenario where there are many NSs downstream of the router. Even
> if the concurrent performance of the router can be improved by increasing the
> number of handlers, it is still relatively slow. More threads will increase
> the CPU context switching time, and in fact many of the handler threads are
> in a blocked state, which is undoubtedly a waste of thread resources. When a
> request enters the router, there is no guarantee that there will be a running
> handler at this time.
>
> Therefore, I consider asynchronous router rpc. Please view the *pdf* for the
> complete solution.
>
> Welcome everyone to exchange and discuss!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17098) DatanodeManager does not handle null storage type properly
[
https://issues.apache.org/jira/browse/HDFS-17098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847888#comment-17847888
]
ASF GitHub Bot commented on HDFS-17098:
---
Hexiaoqiao opened a new pull request, #6840:
URL: https://github.com/apache/hadoop/pull/6840
### Description of PR
1. From https://github.com/apache/hadoop/pull/6035 which contributed by
@teamconfx.
2. Fix checkstyle and try to trigger Yetus again.
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> DatanodeManager does not handle null storage type properly
> --
>
> Key: HDFS-17098
> URL: https://issues.apache.org/jira/browse/HDFS-17098
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ConfX
>Priority: Critical
> Labels: pull-request-available
> Attachments: reproduce.sh
>
>
> h2. What happened:
> Got a {{NullPointerException}} without message when sorting datanodes in
> {{{}NetworkTopology{}}}.
> h2. Where's the bug:
> In line 654 of {{{}DatanodeManager{}}}, the manager creates a second sorter
> using the standard {{Comparator}} class:
> {noformat}
> Comparator comp =
> Comparator.comparing(DatanodeInfoWithStorage::getStorageType);
> secondarySort = list -> Collections.sort(list, comp);{noformat}
> This comparator is then used in {{NetworkTopology}} as a secondary sort to
> break ties:
> {noformat}
> if (secondarySort != null) {
> // a secondary sort breaks the tie between nodes.
> secondarySort.accept(nodesList);
> }{noformat}
> However, if the storage type is {{{}null{}}}, a {{NullPointerException}}
> would be thrown since the default {{Comparator.comparing}} cannot handle
> comparison between null values.
> h2. How to reproduce:
> (1) Set {{dfs.heartbeat.interval}} to {{{}1753310367{}}}, and
> {{dfs.namenode.read.considerStorageType}} to {{true}}
> (2) Run test:
> {{org.apache.hadoop.hdfs.server.blockmanagement.TestSortLocatedBlock#testAviodStaleAndSlowDatanodes}}
> h2. Stacktrace:
> {noformat}
> java.lang.NullPointerException
> at
> java.base/java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469)
> at java.base/java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.base/java.util.TimSort.sort(TimSort.java:220)
> at java.base/java.util.Arrays.sort(Arrays.java:1515)
> at java.base/java.util.ArrayList.sort(ArrayList.java:1750)
> at java.base/java.util.Collections.sort(Collections.java:179)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.lambda$createSecondaryNodeSorter$0(DatanodeManager.java:654)
> at
> org.apache.hadoop.net.NetworkTopology.sortByDistance(NetworkTopology.java:983)
> at
> org.apache.hadoop.net.NetworkTopology.sortByDistanceUsingNetworkLocation(NetworkTopology.java:946)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlock(DatanodeManager.java:637)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:554)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestSortLocatedBlock.testAviodStaleAndSlowDatanodes(TestSortLocatedBlock.java:144){noformat}
> For an easy reproduction, run the reproduce.sh in the attachment. We are
> happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17529) Improve router state store cache entry deletion
[
https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847861#comment-17847861
]
ASF GitHub Bot commented on HDFS-17529:
---
hadoop-yetus commented on PR #6833:
URL: https://github.com/apache/hadoop/pull/6833#issuecomment-2120301224
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 28s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch failed. |
| -1 :x: | compile | 0m 32s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 0m 32s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 28s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-hdfs-rbf in the patch failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 28s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-hdfs-rbf in the patch failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| -1 :x: | mvnsite | 0m 29s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch failed. |
| -1 :x: | javadoc | 0m 28s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6833/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in the
[jira] [Commented] (HDFS-17532) Allow router state store cache update to overwrite and delete in parallel
[
https://issues.apache.org/jira/browse/HDFS-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847846#comment-17847846
]
ASF GitHub Bot commented on HDFS-17532:
---
hadoop-yetus commented on PR #6839:
URL: https://github.com/apache/hadoop/pull/6839#issuecomment-2120145776
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 15m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 2m 6s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 24s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 24s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-hdfs-rbf in trunk failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -0 :warning: | checkstyle | 0m 21s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| The patch fails to run checkstyle in hadoop-hdfs-rbf |
| -1 :x: | mvnsite | 0m 23s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in trunk failed. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in trunk failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-hdfs-rbf in trunk failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 0m 23s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6839/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-hdfs-rbf in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 0m 23s |
[jira] [Commented] (HDFS-17532) Allow router state store cache update to overwrite and delete in parallel
[ https://issues.apache.org/jira/browse/HDFS-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847835#comment-17847835 ] ASF GitHub Bot commented on HDFS-17532: --- kokonguyen191 commented on PR #6839: URL: https://github.com/apache/hadoop/pull/6839#issuecomment-2120093100 @ZanderXu This is the other half split from https://github.com/apache/hadoop/pull/6833, can help me review if you're free, thanks! > Allow router state store cache update to overwrite and delete in parallel > - > > Key: HDFS-17532 > URL: https://issues.apache.org/jira/browse/HDFS-17532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Minor > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket aims to allow the overwrite part and delete part of > org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords > to run in parallel. > See HDFS-17529 for the other half of this improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17532) Allow router state store cache update to overwrite and delete in parallel
[ https://issues.apache.org/jira/browse/HDFS-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847833#comment-17847833 ] ASF GitHub Bot commented on HDFS-17532: --- kokonguyen191 opened a new pull request, #6839: URL: https://github.com/apache/hadoop/pull/6839 ### Description of PR This ticket aims to allow the overwrite part and delete part of org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords to run in parallel. Sister ticket to HDFS-17529 > Allow router state store cache update to overwrite and delete in parallel > - > > Key: HDFS-17532 > URL: https://issues.apache.org/jira/browse/HDFS-17532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Minor > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket aims to allow the overwrite part and delete part of > org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords > to run in parallel. > See HDFS-17529 for the other half of this improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17532) Allow router state store cache update to overwrite and delete in parallel
[ https://issues.apache.org/jira/browse/HDFS-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17532: -- Labels: pull-request-available (was: ) > Allow router state store cache update to overwrite and delete in parallel > - > > Key: HDFS-17532 > URL: https://issues.apache.org/jira/browse/HDFS-17532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Minor > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket aims to allow the overwrite part and delete part of > org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords > to run in parallel. > See HDFS-17529 for the other half of this improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17532) Allow router state store cache update to overwrite and delete in parallel
Felix N created HDFS-17532: -- Summary: Allow router state store cache update to overwrite and delete in parallel Key: HDFS-17532 URL: https://issues.apache.org/jira/browse/HDFS-17532 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs, rbf Reporter: Felix N Assignee: Felix N Current implementation for router state store update is quite inefficient, so much that when routers are removed and a lot of NameNodeMembership records are deleted in a short burst, the deletions triggered a router safemode in our cluster and caused a lot of troubles. This ticket aims to allow the overwrite part and delete part of org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords to run in parallel. See HDFS-17529 for the other half of this improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17529) Improve router state store cache entry deletion
[ https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix N updated HDFS-17529: --- Description: Current implementation for router state store update is quite inefficient, so much that when routers are removed and a lot of NameNodeMembership records are deleted in a short burst, the deletions triggered a router safemode in our cluster and caused a lot of troubles. This ticket aims to improve the deletion process for ZK state store implementation. See HDFS-17532 for the other half of this improvement was: Current implementation for router state store update is quite inefficient, so much that when routers are removed and a lot of NameNodeMembership records are deleted in a short burst, the deletions triggered a router safemode in our cluster and caused a lot of troubles. This ticket aims to improve the deletion process for ZK state store implementation. > Improve router state store cache entry deletion > --- > > Key: HDFS-17529 > URL: https://issues.apache.org/jira/browse/HDFS-17529 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Major > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket aims to improve the deletion process for ZK state store > implementation. > See HDFS-17532 for the other half of this improvement -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17529) Improve router state store cache entry deletion
[ https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix N updated HDFS-17529: --- Description: Current implementation for router state store update is quite inefficient, so much that when routers are removed and a lot of NameNodeMembership records are deleted in a short burst, the deletions triggered a router safemode in our cluster and caused a lot of troubles. This ticket aims to improve the deletion process for ZK state store implementation. was: Current implementation for router state store update is quite inefficient, so much that when routers are removed and a lot of NameNodeMembership records are deleted in a short burst, the deletions triggered a router safemode in our cluster and caused a lot of troubles. This ticket contains 2 parts: improving the deletion process for ZK state store implementation, and allowing the overwrite part and delete part of org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords to run in parallel. > Improve router state store cache entry deletion > --- > > Key: HDFS-17529 > URL: https://issues.apache.org/jira/browse/HDFS-17529 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Major > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket aims to improve the deletion process for ZK state store > implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17529) Improve router state store cache entry deletion
[ https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847816#comment-17847816 ] ASF GitHub Bot commented on HDFS-17529: --- kokonguyen191 commented on PR #6833: URL: https://github.com/apache/hadoop/pull/6833#issuecomment-2120015079 @ZanderXu Thanks for the review, I have updated the codes + changed the ticket/PR title for the deletion part only, will open another PR for the async part later. I'm a bit confused about point 3, can you elaborate a bit on that part? > Improve router state store cache entry deletion > --- > > Key: HDFS-17529 > URL: https://issues.apache.org/jira/browse/HDFS-17529 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Major > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket contains 2 parts: improving the deletion process for ZK state > store implementation, and allowing the overwrite part and delete part of > org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords > to run in parallel. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17529) Improve router state store cache entry deletion
[ https://issues.apache.org/jira/browse/HDFS-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix N updated HDFS-17529: --- Summary: Improve router state store cache entry deletion (was: Improve router state store cache update) > Improve router state store cache entry deletion > --- > > Key: HDFS-17529 > URL: https://issues.apache.org/jira/browse/HDFS-17529 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, rbf >Reporter: Felix N >Assignee: Felix N >Priority: Major > Labels: pull-request-available > > Current implementation for router state store update is quite inefficient, so > much that when routers are removed and a lot of NameNodeMembership records > are deleted in a short burst, the deletions triggered a router safemode in > our cluster and caused a lot of troubles. > This ticket contains 2 parts: improving the deletion process for ZK state > store implementation, and allowing the overwrite part and delete part of > org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore#overrideExpiredRecords > to run in parallel. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
