[jira] [Commented] (HADOOP-17153) Add boost installation steps to build instruction on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165128#comment-17165128 ] Hudson commented on HADOOP-17153: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18471 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18471/]) HADOOP-17153. Add boost installation steps to build instruction on (github: rev 4b1816c7d0188363896505d0f0f93cb58d44bcd9) * (edit) BUILDING.txt > Add boost installation steps to build instruction on CentOS 8 > - > > Key: HADOOP-17153 > URL: https://issues.apache.org/jira/browse/HADOOP-17153 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.4.0 > > > After HDFS-15385, -Pnative build fails without boost 1.72 used in libhdfs++. > It must be installed from source since boost 1.66 packaged by the CentOS > distribution does not match. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17153) Add boost installation steps to build instruction on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-17153: -- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Add boost installation steps to build instruction on CentOS 8 > - > > Key: HADOOP-17153 > URL: https://issues.apache.org/jira/browse/HADOOP-17153 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.4.0 > > > After HDFS-15385, -Pnative build fails without boost 1.72 used in libhdfs++. > It must be installed from source since boost 1.66 packaged by the CentOS > distribution does not match. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2169: HADOOP-17153. Add boost installation steps to build instruction on CentOS 8.
iwasakims commented on pull request #2169: URL: https://github.com/apache/hadoop/pull/2169#issuecomment-663908887 Thanks, @aajisaka. I merged this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims merged pull request #2169: HADOOP-17153. Add boost installation steps to build instruction on CentOS 8.
iwasakims merged pull request #2169: URL: https://github.com/apache/hadoop/pull/2169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on pull request #2166: HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids.
mukul1987 commented on pull request #2166: URL: https://github.com/apache/hadoop/pull/2166#issuecomment-663883813 @bshashikant can you please paste the command output in the PR pelase ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #2165: HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots
mukul1987 commented on a change in pull request #2165:
URL: https://github.com/apache/hadoop/pull/2165#discussion_r460428877
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotDeletionGc.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.snapshot;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static
org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.DFS_NAMENODE_SNAPSHOT_DELETION_ORDERED_GC_PERIOD_MS;
+import static
org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.DFS_NAMENODE_SNAPSHOT_DELETION_ORDERED_GC_PERIOD_MS_DEFAULT;
+
+public class SnapshotDeletionGc {
+ public static final Logger LOG = LoggerFactory.getLogger(
+ SnapshotDeletionGc.class);
+
+ private final FSNamesystem namesystem;
+ private final int deletionOrderedGcPeriodMs;
+ private final AtomicReference timer = new AtomicReference<>();
+
+ public SnapshotDeletionGc(FSNamesystem namesystem, Configuration conf) {
+this.namesystem = namesystem;
+
+this.deletionOrderedGcPeriodMs = conf.getInt(
+DFS_NAMENODE_SNAPSHOT_DELETION_ORDERED_GC_PERIOD_MS,
+DFS_NAMENODE_SNAPSHOT_DELETION_ORDERED_GC_PERIOD_MS_DEFAULT);
+LOG.info("{} = {}", DFS_NAMENODE_SNAPSHOT_DELETION_ORDERED_GC_PERIOD_MS,
+deletionOrderedGcPeriodMs);
+ }
+
+ public void schedule() {
+if (timer.get() != null) {
+ return;
+}
+final Timer t = new Timer(getClass().getSimpleName(), true);
+if (timer.compareAndSet(null, t)) {
+ LOG.info("Schedule at fixed rate {}",
+ StringUtils.formatTime(deletionOrderedGcPeriodMs));
+ t.scheduleAtFixedRate(new GcTask(),
+ deletionOrderedGcPeriodMs, deletionOrderedGcPeriodMs);
+}
+ }
+
+ public void cancel() {
+final Timer t = timer.getAndSet(null);
+if (t != null) {
+ LOG.info("cancel");
+ t.cancel();
+}
+ }
+
+ private void gcDeletedSnapshot(String name) {
+final Snapshot.Root deleted;
+namesystem.readLock();
Review comment:
Lets add an assert that this is the first snapshot in the snapshottable
dir ?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##
@@ -629,4 +641,36 @@ public void shutdown() {
s.getRoot().getLocalName(), s.getRoot().getFullPathName(),
s.getRoot().getModificationTime());
}
-}
+
+ Snapshot.Root chooseDeletedSnapshot() {
+final List dirs = getSnapshottableDirs();
+Collections.shuffle(dirs);
Review comment:
Lets move this to a different funcation like chooseSnapshottableDir, as
this might be policy controleed later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2171: YARN-10367. Failed to get nodejs 10.21.0 when building docker image
hadoop-yetus commented on pull request #2171: URL: https://github.com/apache/hadoop/pull/2171#issuecomment-663864658 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 30m 46s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 1m 11s | Maven dependency ordering for branch | | +1 :green_heart: | shadedclient | 15m 36s | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch | | -1 :x: | hadolint | 0m 4s | The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | shelldocs | 0m 14s | There were no new shelldocs issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 25s | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 65m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2171/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2171 | | Optional Tests | dupname asflicense hadolint shellcheck shelldocs | | uname | Linux 54e6f5c1751c 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ac5f21dbef0 | | hadolint | https://builds.apache.org/job/hadoop-multibranch/job/PR-2171/1/artifact/out/diff-patch-hadolint.txt | | Max. process+thread count | 338 (vs. ulimit of 5500) | | modules | C: U: | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2171/1/console | | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 hadolint=1.11.1-0-g0e692dd | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17132) ABFS: Fix For Idempotency code
[
https://issues.apache.org/jira/browse/HADOOP-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17132:
-
Fix Version/s: 3.3.1
1ac5258 branch-3.3@\{2020-07-25 13:13:05 +}: cherry-pick: Hadoop 17132.
ABFS: Fix Rename and Delete Idempotency check trigger
> ABFS: Fix For Idempotency code
> --
>
> Key: HADOOP-17132
> URL: https://issues.apache.org/jira/browse/HADOOP-17132
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.4.0
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Labels: abfsactive
> Fix For: 3.3.1, 3.4.0
>
>
> Trigger to handle the idempotency code introduced in
> https://issues.apache.org/jira/browse/HADOOP-17015 is incomplete.
> This PR is to fix the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17058) Support for Appendblob in abfs driver
[
https://issues.apache.org/jira/browse/HADOOP-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17058:
-
Fix Version/s: 3.3.1
f24e2ec branch-3.3@\{2020-07-25 13:12:32 +}: cherry-pick: HADOOP-17058.
ABFS: Support for AppendBlob in Hadoop ABFS Driver
> Support for Appendblob in abfs driver
> -
>
> Key: HADOOP-17058
> URL: https://issues.apache.org/jira/browse/HADOOP-17058
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Ishani
>Assignee: Ishani
>Priority: Major
> Fix For: 3.3.1
>
>
> add changes to support appendblob in the hadoop-azure abfs driver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16961) ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
[
https://issues.apache.org/jira/browse/HADOOP-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-16961:
-
Fix Version/s: 3.3.1
7c9b459 branch-3.3@\{2020-07-25 13:12:09 +}: cherry-pick: HADOOP-16961.
ABFS: Adding metrics to AbfsInputStream (#2076)
> ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
> ---
>
> Key: HADOOP-16961
> URL: https://issues.apache.org/jira/browse/HADOOP-16961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Mehakmeet Singh
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the
> testing and diagnostics of the connector.
> Also adding some logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17065) Adding Network Counters in ABFS
[
https://issues.apache.org/jira/browse/HADOOP-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17065:
-
Fix Version/s: 3.3.1
0d0ed17 branch-3.3@\{2020-07-25 13:11:10 +}: cherry-pick: HADOOP-17065. Add
Network Counters to ABFS (#2056)
> Adding Network Counters in ABFS
> ---
>
> Key: HADOOP-17065
> URL: https://issues.apache.org/jira/browse/HADOOP-17065
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Fix For: 3.3.1
>
>
> Network Counters to be added in ABFS:
> |CONNECTIONS_MADE|Number of times connection was made with Azure Data Lake|
> |SEND_REQUESTS|Number of send requests|
> |GET_RESPONSE|Number of response gotten|
> |BYTES_SEND|Number of bytes send|
> |BYTES_RECEIVED|Number of bytes received|
> |READ_THROTTLE|Number of times throttled while read operation|
> |WRITE_THROTTLE|Number of times throttled while write operation|
> propose:
> * Adding these counters as part of AbfsStatistic already made in
> HADOOP-17016.
> * Increment of counters across Abfs Network services.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17057) ABFS driver enhancement - Allow customizable translation from AAD SPNs and security groups to Linux user and group
[
https://issues.apache.org/jira/browse/HADOOP-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17057:
-
Fix Version/s: 3.3.1
8b7e774 branch-3.3@\{2020-07-25 13:10:39 +}: cherry-pick: HDFS-15168: ABFS
enhancement to translate AAD to Linux identities. (#1978)
> ABFS driver enhancement - Allow customizable translation from AAD SPNs and
> security groups to Linux user and group
> --
>
> Key: HADOOP-17057
> URL: https://issues.apache.org/jira/browse/HADOOP-17057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Reporter: Karthik Amarnath
>Assignee: Karthik Amarnath
>Priority: Major
> Fix For: 3.3.1
>
>
> ABFS driver does not support the translation of AAD Service principal (SPI)
> to Linux identities causing metadata operation failure. Hadoop MapReduce
> client
> [[JobSubmissionFiles|https://github.com/apache/hadoop/blob/d842dfffa53c8b565f3d65af44ccd7e1cc706733/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L138]]
> expects the file owner permission to be the Linux identity, but the
> underlying ABFS driver returns the AAD Object identity. Hence need ABFS
> driver enhancement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17053) ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting
[
https://issues.apache.org/jira/browse/HADOOP-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17053:
-
903935d branch-3.3@\{2020-07-25 13:10:30 +}: cherry-pick: HADOOP-17053.
ABFS: Fix Account-specific OAuth config setting parsing
> ABFS: FS initialize fails for incompatible account-agnostic Token Provider
> setting
> ---
>
> Key: HADOOP-17053
> URL: https://issues.apache.org/jira/browse/HADOOP-17053
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> When AuthType and Auth token provider configs are set for both generic and
> account specific config, as below:
> // account agnostic
> fs.azure.account.auth.type=CUSTOM
> fs.azure.account.oauth.provider.type=ClassExtendingCustomTokenProviderAdapter
> // account specific
> fs.azure.account.auth.type.account_name=OAuth
> fs.azure.account.oauth.provider.type.account_name=ClassExtendingAccessTokenProvider
> For account_name, OAuth with provider as ClassExtendingAccessTokenProvider
> is expected to be in effect.
> When the token provider class is being read from the config, account agnostic
> config setting is read first in the assumption that it can serve as default
> if account-specific config setting is absent. But this logic leads to failure
> when AuthType set for account specific and otherwise are different as the
> Interface implementing the token provider is different for various Auth
> Types. This leads to a Runtime exception when trying to create the oAuth
> access token provider.
> This Jira is to track the fix for it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17053) ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting
[
https://issues.apache.org/jira/browse/HADOOP-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17053:
-
Fix Version/s: 3.3.1
903935d branch-3.3@\{2020-07-25 13:10:30 +}: cherry-pick: HADOOP-17053.
ABFS: Fix Account-specific OAuth config setting parsing
> ABFS: FS initialize fails for incompatible account-agnostic Token Provider
> setting
> ---
>
> Key: HADOOP-17053
> URL: https://issues.apache.org/jira/browse/HADOOP-17053
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> When AuthType and Auth token provider configs are set for both generic and
> account specific config, as below:
> // account agnostic
> fs.azure.account.auth.type=CUSTOM
> fs.azure.account.oauth.provider.type=ClassExtendingCustomTokenProviderAdapter
> // account specific
> fs.azure.account.auth.type.account_name=OAuth
> fs.azure.account.oauth.provider.type.account_name=ClassExtendingAccessTokenProvider
> For account_name, OAuth with provider as ClassExtendingAccessTokenProvider
> is expected to be in effect.
> When the token provider class is being read from the config, account agnostic
> config setting is read first in the assumption that it can serve as default
> if account-specific config setting is absent. But this logic leads to failure
> when AuthType set for account specific and otherwise are different as the
> Interface implementing the token provider is different for various Auth
> Types. This leads to a Runtime exception when trying to create the oAuth
> access token provider.
> This Jira is to track the fix for it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16852) ABFS: Send error back to client for Read Ahead request failure
[
https://issues.apache.org/jira/browse/HADOOP-16852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-16852:
-
Fix Version/s: 3.3.1
869a68b branch-3.3@\{2020-07-25 13:10:19 +}: cherry-pick: HADOOP-16852:
Report read-ahead error bac
> ABFS: Send error back to client for Read Ahead request failure
> --
>
> Key: HADOOP-16852
> URL: https://issues.apache.org/jira/browse/HADOOP-16852
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1
>
>
> Issue seen by a customer:
> The failed requests we were seeing in the AbfsClient logging actually never
> made it out over the wire. We have found that there’s an issue with ADLS
> passthrough and the 8 read ahead threads that ADLSv2 spawns in
> ReadBufferManager.java. We depend on thread local storage in order to get the
> right JWT token and those threads do not have the right information in their
> thread local storage. Thus, when they pick up a task from the read ahead
> queue they fail by throwing an AzureCredentialNotFoundException exception in
> AbfsRestOperation.executeHttpOperation() where it calls
> client.getAccessToken(). This exception is silently swallowed by the read
> ahead threads in ReadBufferWorker.run(). As a result, every read ahead
> attempt results in a failed executeHttpOperation(), but still calls
> AbfsClientThrottlingIntercept.updateMetrics() and contributes to throttling
> (despite not making it out over the wire). After the read aheads fail, the
> main task thread performs the read with the right thread local storage
> information and succeeds, but first sleeps for up to 10 seconds due to the
> throttling.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17054) ABFS: Fix idempotency test failures when SharedKey is set as AuthType
[
https://issues.apache.org/jira/browse/HADOOP-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17054:
-
Fix Version/s: 3.3.1
27b20f9 branch-3.3@\{2020-07-25 13:09:26 +}: cherry-pick: HADOOP-17054.
ABFS: Fix test AbfsClient authentication instance
> ABFS: Fix idempotency test failures when SharedKey is set as AuthType
> -
>
> Key: HADOOP-17054
> URL: https://issues.apache.org/jira/browse/HADOOP-17054
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> Idempotency related tests added as part of
> https://issues.apache.org/jira/browse/HADOOP-17015
> create a test AbfsClient instance. This mock instance wrongly accepts valid
> sharedKey and oauth token provider instance. This leads to test failures with
> exceptions:
> [ERROR]
> testRenameRetryFailureAsHTTP404(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename)
> Time elapsed: 9.133 s <<< ERROR!
> Invalid auth type: SharedKey is being used, expecting OAuth
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:643)
> This Jira is to fix these tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17015) ABFS: Make PUT and POST operations idempotent
[
https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17015:
-
Fix Version/s: 3.3.1
eed06b4 branch-3.3@\{2020-07-25 13:08:01 +}: cherry-pick: Hadoop-17015.
ABFS: Handling Rename and Delete idempotency
> ABFS: Make PUT and POST operations idempotent
> -
>
> Key: HADOOP-17015
> URL: https://issues.apache.org/jira/browse/HADOOP-17015
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> Currently when a PUT or POST operation timeouts and the server has already
> successfully executed the operation, there is no check in driver to see if
> the operation did succeed or not and just retries the same operation again.
> This can cause driver to through invalid user errors.
>
> Sample scenario:
> # Rename request times out. Though server has successfully executed the
> operation.
> # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success
> if source if not found, but destination is present.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2171: YARN-10367. Failed to get nodejs 10.21.0 when building docker image
hadoop-yetus commented on pull request #2171: URL: https://github.com/apache/hadoop/pull/2171#issuecomment-663857901 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-2171/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #2171: YARN-10367. Failed to get nodejs 10.21.0 when building docker image
aajisaka opened a new pull request #2171: URL: https://github.com/apache/hadoop/pull/2171 JIRA: https://issues.apache.org/jira/browse/YARN-10367 Install the latest 10.x version of nodejs in the docker image. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17145) Unauthenticated users are not authorized to access this page message is misleading in HttpServer2.java
[
https://issues.apache.org/jira/browse/HADOOP-17145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164928#comment-17164928
]
Hadoop QA commented on HADOOP-17145:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 34m
2s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-17145 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008338/HADOOP-17145.002.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/5/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Unauthenticated users are not authorized to access this page message is
> misleading in HttpServer2.java
> --
>
> Key: HADOOP-17145
> URL: https://issues.apache.org/jira/browse/HADOOP-17145
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-17145.001.patch, HADOOP-17145.002.patch
>
>
> Recently one of the users were misled by the message "Unauthenticated users
> are not authorized to access this page" when the user was not an admin user.
> At that point the user is authenticated but has no admin access, so it's
> actually not an authentication issue but an authorization issue.
> Also, 401 as error code would be better.
> Something like "User is unauthorized to access the page" would help to users
> to find out what is the problem during access an http endpoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17145) Unauthenticated users are not authorized to access this page message is misleading in HttpServer2.java
[
https://issues.apache.org/jira/browse/HADOOP-17145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164925#comment-17164925
]
Hadoop QA commented on HADOOP-17145:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 35m
48s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-17145 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008338/HADOOP-17145.002.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/2/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Unauthenticated users are not authorized to access this page message is
> misleading in HttpServer2.java
> --
>
> Key: HADOOP-17145
> URL: https://issues.apache.org/jira/browse/HADOOP-17145
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-17145.001.patch, HADOOP-17145.002.patch
>
>
> Recently one of the users were misled by the message "Unauthenticated users
> are not authorized to access this page" when the user was not an admin user.
> At that point the user is authenticated but has no admin access, so it's
> actually not an authentication issue but an authorization issue.
> Also, 401 as error code would be better.
> Something like "User is unauthorized to access the page" would help to users
> to find out what is the problem during access an http endpoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17151) upgrade jetty to 9.4.21
[
https://issues.apache.org/jira/browse/HADOOP-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164926#comment-17164926
]
Hadoop QA commented on HADOOP-17151:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 35m
48s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-17151 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008264/HADOOP-17151.001.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> upgrade jetty to 9.4.21
> ---
>
> Key: HADOOP-17151
> URL: https://issues.apache.org/jira/browse/HADOOP-17151
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.3.0
>Reporter: liusheng
>Priority: Major
> Attachments: HADOOP-17151.001.patch
>
>
> I have tried to configure and start Hadoop KMS service, it was failed to
> start the error log messages:
> {noformat}
> 2020-07-23 10:57:31,872 INFO Server - jetty-9.4.20.v20190813; built:
> 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm
> 1.8.0_252-8u252-b09-1~18.04-b09
> 2020-07-23 10:57:31,899 INFO session - DefaultSessionIdManager
> workerName=node0
> 2020-07-23 10:57:31,899 INFO session - No SessionScavenger set, using
> defaults
> 2020-07-23 10:57:31,901 INFO session - node0 Scavenging every 66ms
> 2020-07-23 10:57:31,912 INFO ContextHandler - Started
> o.e.j.s.ServletContextHandler@5bf0d49{logs,/logs,file:///opt/hadoop-3.4.0-SNAPSHOT/logs/,AVAILABLE}
> 2020-07-23 10:57:31,913 INFO ContextHandler - Started
> o.e.j.s.ServletContextHandler@7c7a06ec{static,/static,jar:file:/opt/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/hadoop-kms-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2020-07-23 10:57:31,986 INFO TypeUtil - JVM Runtime does not support Modules
> 2020-07-23 10:57:32,015 INFO KMSWebApp -
> -
> 2020-07-23 10:57:32,015 INFO KMSWebApp - Java runtime version :
> 1.8.0_252-8u252-b09-1~18.04-b09
> 2020-07-23 10:57:32,015 INFO KMSWebApp - User: hadoop
> 2020-07-23 10:57:32,015 INFO KMSWebApp - KMS Hadoop Version: 3.4.0-SNAPSHOT
> 2020-07-23 10:57:32,015 INFO KMSWebApp -
> -
> 2020-07-23 10:57:32,023 INFO KMSACLs - 'CREATE' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'DELETE' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'ROLLOVER' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET_KEYS' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET_METADATA' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'SET_KEY_MATERIAL' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GENERATE_EEK' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'DECRYPT_EEK' ACL '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP 'READ' is
> set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'MANAGEMENT' is set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'GENERATE_EEK' is set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'DECRYPT_EEK' is set to '*'
> 2020-07-23 10:57:32,080 INFO KMSAudit - Initializing audit logger class
> org.apache.hadoop.crypto.key.kms.server.SimpleKMSAuditLogger
> 2020-07-23 10:57:32,537 INFO KMSWebServer - SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down KMSWebServer at
> hadoop-benchmark/172.17.0.2{noformat}
> I have googled the error and found there is a simlar issue：
> [https://github.com/eclipse/jetty.project/issues/4064]
> It looks like a bug of jetty and has been fixed in jetty>=9.4.21, currently
> Hadoop use the jetty is version of 9.4.20, see hadoop-project/pom.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17151) upgrade jetty to 9.4.21
[
https://issues.apache.org/jira/browse/HADOOP-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164922#comment-17164922
]
Hadoop QA commented on HADOOP-17151:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 26m
40s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-17151 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008264/HADOOP-17151.001.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/4/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> upgrade jetty to 9.4.21
> ---
>
> Key: HADOOP-17151
> URL: https://issues.apache.org/jira/browse/HADOOP-17151
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.3.0
>Reporter: liusheng
>Priority: Major
> Attachments: HADOOP-17151.001.patch
>
>
> I have tried to configure and start Hadoop KMS service, it was failed to
> start the error log messages:
> {noformat}
> 2020-07-23 10:57:31,872 INFO Server - jetty-9.4.20.v20190813; built:
> 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm
> 1.8.0_252-8u252-b09-1~18.04-b09
> 2020-07-23 10:57:31,899 INFO session - DefaultSessionIdManager
> workerName=node0
> 2020-07-23 10:57:31,899 INFO session - No SessionScavenger set, using
> defaults
> 2020-07-23 10:57:31,901 INFO session - node0 Scavenging every 66ms
> 2020-07-23 10:57:31,912 INFO ContextHandler - Started
> o.e.j.s.ServletContextHandler@5bf0d49{logs,/logs,file:///opt/hadoop-3.4.0-SNAPSHOT/logs/,AVAILABLE}
> 2020-07-23 10:57:31,913 INFO ContextHandler - Started
> o.e.j.s.ServletContextHandler@7c7a06ec{static,/static,jar:file:/opt/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/hadoop-kms-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2020-07-23 10:57:31,986 INFO TypeUtil - JVM Runtime does not support Modules
> 2020-07-23 10:57:32,015 INFO KMSWebApp -
> -
> 2020-07-23 10:57:32,015 INFO KMSWebApp - Java runtime version :
> 1.8.0_252-8u252-b09-1~18.04-b09
> 2020-07-23 10:57:32,015 INFO KMSWebApp - User: hadoop
> 2020-07-23 10:57:32,015 INFO KMSWebApp - KMS Hadoop Version: 3.4.0-SNAPSHOT
> 2020-07-23 10:57:32,015 INFO KMSWebApp -
> -
> 2020-07-23 10:57:32,023 INFO KMSACLs - 'CREATE' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'DELETE' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'ROLLOVER' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET_KEYS' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET_METADATA' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'SET_KEY_MATERIAL' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GENERATE_EEK' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'DECRYPT_EEK' ACL '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP 'READ' is
> set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'MANAGEMENT' is set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'GENERATE_EEK' is set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'DECRYPT_EEK' is set to '*'
> 2020-07-23 10:57:32,080 INFO KMSAudit - Initializing audit logger class
> org.apache.hadoop.crypto.key.kms.server.SimpleKMSAuditLogger
> 2020-07-23 10:57:32,537 INFO KMSWebServer - SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down KMSWebServer at
> hadoop-benchmark/172.17.0.2{noformat}
> I have googled the error and found there is a simlar issue：
> [https://github.com/eclipse/jetty.project/issues/4064]
> It looks like a bug of jetty and has been fixed in jetty>=9.4.21, currently
> Hadoop use the jetty is version of 9.4.20, see hadoop-project/pom.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17126) implement non-guava Precondition checkNotNull
[
https://issues.apache.org/jira/browse/HADOOP-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164921#comment-17164921
]
Hadoop QA commented on HADOOP-17126:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 26m
30s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-17126 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008191/HADOOP-17126.001.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/6/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> implement non-guava Precondition checkNotNull
> -
>
> Key: HADOOP-17126
> URL: https://issues.apache.org/jira/browse/HADOOP-17126
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HADOOP-17126.001.patch
>
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to create {{checkNotNull}} in a new package dubbed {{unguava}}.
> +The plan is as follows+
> * create a new {{package org.apache.hadoop.util.unguava;}}
> * {{create class Validate}}
> * implement {{package org.apache.hadoop.util.unguava.Validate;}} with the
> following interface
> ** {{checkNotNull(final T obj)}}
> ** {{checkNotNull(final T reference, final Object errorMessage)}}
> ** {{checkNotNull(final T obj, final String message, final Object...
> values)}}
> ** {{checkNotNull(final T obj,final Supplier msgSupplier)}}
> * {{guava.preconditions used String.lenientformat which suppressed
> exceptions caused by string formatting of the exception message . So, in
> order to avoid changing the behavior, the implementation catches Exceptions
> triggered by building the message (IllegalFormat, InsufficientArg,
> NullPointer..etc)}}
> * {{After merging the new class, we can replace
> guava.Preconditions.checkNotNull by {{unguava.Validate.checkNotNull
> * We need the change to go into trunk, 3.1, 3.2, and 3.3
>
> Similar Jiras will be created to implement checkState, checkArgument,
> checkIndex
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17126) implement non-guava Precondition checkNotNull
[
https://issues.apache.org/jira/browse/HADOOP-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164920#comment-17164920
]
Hadoop QA commented on HADOOP-17126:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 28m
7s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-17126 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008191/HADOOP-17126.001.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/3/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> implement non-guava Precondition checkNotNull
> -
>
> Key: HADOOP-17126
> URL: https://issues.apache.org/jira/browse/HADOOP-17126
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HADOOP-17126.001.patch
>
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to create {{checkNotNull}} in a new package dubbed {{unguava}}.
> +The plan is as follows+
> * create a new {{package org.apache.hadoop.util.unguava;}}
> * {{create class Validate}}
> * implement {{package org.apache.hadoop.util.unguava.Validate;}} with the
> following interface
> ** {{checkNotNull(final T obj)}}
> ** {{checkNotNull(final T reference, final Object errorMessage)}}
> ** {{checkNotNull(final T obj, final String message, final Object...
> values)}}
> ** {{checkNotNull(final T obj,final Supplier msgSupplier)}}
> * {{guava.preconditions used String.lenientformat which suppressed
> exceptions caused by string formatting of the exception message . So, in
> order to avoid changing the behavior, the implementation catches Exceptions
> triggered by building the message (IllegalFormat, InsufficientArg,
> NullPointer..etc)}}
> * {{After merging the new class, we can replace
> guava.Preconditions.checkNotNull by {{unguava.Validate.checkNotNull
> * We need the change to go into trunk, 3.1, 3.2, and 3.3
>
> Similar Jiras will be created to implement checkState, checkArgument,
> checkIndex
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17154) Failed to get nodejs 10.21.0 when building docker image
Akira Ajisaka created HADOOP-17154:
--
Summary: Failed to get nodejs 10.21.0 when building docker image
Key: HADOOP-17154
URL: https://issues.apache.org/jira/browse/HADOOP-17154
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Akira Ajisaka
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/213/console
{noformat}
Version '10.21.0-1nodesource1' for 'nodejs' was not found
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2169: HADOOP-17153. Add boost installation steps to build instruction on CentOS 8.
aajisaka commented on pull request #2169: URL: https://github.com/apache/hadoop/pull/2169#issuecomment-663854559 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17151) upgrade jetty to 9.4.21
[
https://issues.apache.org/jira/browse/HADOOP-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164762#comment-17164762
]
liusheng commented on HADOOP-17151:
---
Hi [~ste...@apache.org],
Yes, I have tested locally, doing this can fix the problem.
> upgrade jetty to 9.4.21
> ---
>
> Key: HADOOP-17151
> URL: https://issues.apache.org/jira/browse/HADOOP-17151
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.3.0
>Reporter: liusheng
>Priority: Major
> Attachments: HADOOP-17151.001.patch
>
>
> I have tried to configure and start Hadoop KMS service, it was failed to
> start the error log messages:
> {noformat}
> 2020-07-23 10:57:31,872 INFO Server - jetty-9.4.20.v20190813; built:
> 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm
> 1.8.0_252-8u252-b09-1~18.04-b09
> 2020-07-23 10:57:31,899 INFO session - DefaultSessionIdManager
> workerName=node0
> 2020-07-23 10:57:31,899 INFO session - No SessionScavenger set, using
> defaults
> 2020-07-23 10:57:31,901 INFO session - node0 Scavenging every 66ms
> 2020-07-23 10:57:31,912 INFO ContextHandler - Started
> o.e.j.s.ServletContextHandler@5bf0d49{logs,/logs,file:///opt/hadoop-3.4.0-SNAPSHOT/logs/,AVAILABLE}
> 2020-07-23 10:57:31,913 INFO ContextHandler - Started
> o.e.j.s.ServletContextHandler@7c7a06ec{static,/static,jar:file:/opt/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/hadoop-kms-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2020-07-23 10:57:31,986 INFO TypeUtil - JVM Runtime does not support Modules
> 2020-07-23 10:57:32,015 INFO KMSWebApp -
> -
> 2020-07-23 10:57:32,015 INFO KMSWebApp - Java runtime version :
> 1.8.0_252-8u252-b09-1~18.04-b09
> 2020-07-23 10:57:32,015 INFO KMSWebApp - User: hadoop
> 2020-07-23 10:57:32,015 INFO KMSWebApp - KMS Hadoop Version: 3.4.0-SNAPSHOT
> 2020-07-23 10:57:32,015 INFO KMSWebApp -
> -
> 2020-07-23 10:57:32,023 INFO KMSACLs - 'CREATE' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'DELETE' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'ROLLOVER' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET_KEYS' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GET_METADATA' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'SET_KEY_MATERIAL' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'GENERATE_EEK' ACL '*'
> 2020-07-23 10:57:32,024 INFO KMSACLs - 'DECRYPT_EEK' ACL '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP 'READ' is
> set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'MANAGEMENT' is set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'GENERATE_EEK' is set to '*'
> 2020-07-23 10:57:32,025 INFO KMSACLs - default.key.acl. for KEY_OP
> 'DECRYPT_EEK' is set to '*'
> 2020-07-23 10:57:32,080 INFO KMSAudit - Initializing audit logger class
> org.apache.hadoop.crypto.key.kms.server.SimpleKMSAuditLogger
> 2020-07-23 10:57:32,537 INFO KMSWebServer - SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down KMSWebServer at
> hadoop-benchmark/172.17.0.2{noformat}
> I have googled the error and found there is a simlar issue：
> [https://github.com/eclipse/jetty.project/issues/4064]
> It looks like a bug of jetty and has been fixed in jetty>=9.4.21, currently
> Hadoop use the jetty is version of 9.4.20, see hadoop-project/pom.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
