[jira] [Updated] (HADOOP-14508) TestDFSIO throws NPE when set -sequential argument.
[
https://issues.apache.org/jira/browse/HADOOP-14508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HADOOP-14508:
---
Status: Open (was: Patch Available)
-seq seems trivial, so cancel patch.
> TestDFSIO throws NPE when set -sequential argument.
> ---
>
> Key: HADOOP-14508
> URL: https://issues.apache.org/jira/browse/HADOOP-14508
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Wenxin He
>Assignee: Wenxin He
>Priority: Major
> Attachments: HADOOP-14508.002.patch
>
>
> Benchmark tool TestDFSIO throws NPE when set {{-sequential}} due to
> uninitialized {{ioer.stream}} in {{TestDFSIO#sequentialTest}}.
> More descriptions, stack traces see comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS
DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr
support to WASB and ABFS
URL: https://github.com/apache/hadoop/pull/452#discussion_r351542440
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
##
@@ -117,6 +124,70 @@ public void testStoreRetrieveFile() throws Exception {
fs.delete(testFile, true);
}
+ @Test
+ public void testSetGetXAttr() throws Exception {
+byte[] attributeValue1 = "hi".getBytes(StandardCharsets.UTF_8);
+byte[] attributeValue2 = "你好".getBytes(StandardCharsets.UTF_8);
+String attributeName1 = "user.asciiAttribute";
+String attributeName2 = "user.unicodeAttribute";
+Path testFile = methodPath();
+
+// after creating a file, the xAttr should not be present
+createEmptyFile(testFile,
FsPermission.createImmutable(READ_WRITE_PERMISSIONS));
+assertNull(fs.getXAttr(testFile, attributeName1));
+
+// after setting the xAttr on the file, the value should be retrievable
+fs.setXAttr(testFile, attributeName1, attributeValue1);
+assertArrayEquals(attributeValue1, fs.getXAttr(testFile, attributeName1));
+
+// after setting a second xAttr on the file, the first xAttr values should
not be overwritten
+fs.setXAttr(testFile, attributeName2, attributeValue2);
+assertArrayEquals(attributeValue1, fs.getXAttr(testFile, attributeName1));
+assertArrayEquals(attributeValue2, fs.getXAttr(testFile, attributeName2));
+ }
+
+ @Test
+ public void testSetGetXAttrCreateReplace() throws Exception {
+byte[] attributeValue = "one".getBytes(StandardCharsets.UTF_8);
+String attributeName = "user.someAttribute";
+Path testFile = methodPath();
+
+// after creating a file, it must be possible to create a new xAttr
+createEmptyFile(testFile,
FsPermission.createImmutable(READ_WRITE_PERMISSIONS));
+fs.setXAttr(testFile, attributeName, attributeValue, CREATE_FLAG);
+assertArrayEquals(attributeValue, fs.getXAttr(testFile, attributeName));
+
+// however after the xAttr is created, creating it again must fail
+try {
+ fs.setXAttr(testFile, attributeName, attributeValue, CREATE_FLAG);
+ fail("Creating an existing xAttr should fail");
Review comment:
you can use "org.apache.hadoop.test.LambdaTestUtils.intercept" for exception
check, here and in other files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS
DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS URL: https://github.com/apache/hadoop/pull/452#discussion_r351542583 ## File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAttributes.java ## @@ -0,0 +1,112 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.XAttrSetFlag; +import org.junit.Assume; +import org.junit.Test; + +import java.io.IOException; Review comment: import order This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS
DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS URL: https://github.com/apache/hadoop/pull/452#discussion_r351542962 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java ## @@ -632,6 +634,85 @@ public void setOwner(final Path path, final String owner, final String group) } } + /** + * Set the value of an attribute for a path. + * + * @param path The path on which to set the attribute + * @param name The attribute to set + * @param value The byte value of the attribute to set (encoded in latin-1) + * @param flag The mode in which to set the attribute + * @throws IOException If there was an issue setting the attribute on Azure + * @throws IllegalArgumentException If name is null or empty or if value is null + */ + @Override + public void setXAttr(Path path, String name, byte[] value, EnumSet flag) Review comment: it is better to use final for parameter when you can This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS
DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr
support to WASB and ABFS
URL: https://github.com/apache/hadoop/pull/452#discussion_r351541785
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##
@@ -632,6 +634,85 @@ public void setOwner(final Path path, final String owner,
final String group)
}
}
+ /**
+ * Set the value of an attribute for a path.
+ *
+ * @param path The path on which to set the attribute
+ * @param name The attribute to set
+ * @param value The byte value of the attribute to set (encoded in latin-1)
+ * @param flag The mode in which to set the attribute
+ * @throws IOException If there was an issue setting the attribute on Azure
+ * @throws IllegalArgumentException If name is null or empty or if value is
null
+ */
+ @Override
+ public void setXAttr(Path path, String name, byte[] value,
EnumSet flag)
+ throws IOException {
+LOG.debug(
+"AzureBlobFileSystem.setXAttr path: {}", path);
+
+if (name == null || name.isEmpty() || value == null) {
+ throw new IllegalArgumentException("A valid name and value must be
specified.");
+}
+
+Path qualifiedPath = makeQualified(path);
+performAbfsAuthCheck(FsAction.READ_WRITE, qualifiedPath);
+
+try {
+ Hashtable properties = abfsStore.getPathStatus(path);
+ String xAttrName = ensureValidAttributeName(name);
+ boolean xAttrExists = properties.containsKey(xAttrName);
+ XAttrSetFlag.validate(name, xAttrExists, flag);
+
+ String xAttrValue = new String(value,
AzureBlobFileSystemStore.XMS_PROPERTIES_ENCODING);
+ properties.put(xAttrName, xAttrValue);
+ abfsStore.setPathProperties(path, properties);
+} catch (AzureBlobFileSystemException ex) {
+ checkException(path, ex);
+}
+ }
+
+ /**
+ * Get the value of an attribute for a path.
+ *
+ * @param path The path on which to get the attribute
+ * @param name The attribute to get
+ * @return The bytes of the attribute's value (encoded in latin-1)
+ * or null if the attribute does not exist
+ * @throws IOException If there was an issue getting the attribute from Azure
+ * @throws IllegalArgumentException If name is null or empty
+ */
+ @Override
+ public byte[] getXAttr(Path path, String name)
+ throws IOException {
+LOG.debug(
+"AzureBlobFileSystem.getXAttr path: {}", path);
Review comment:
remove line break
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr support to WASB and ABFS
DadanielZ commented on a change in pull request #452: HADOOP-16005: Add XAttr
support to WASB and ABFS
URL: https://github.com/apache/hadoop/pull/452#discussion_r351541745
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##
@@ -632,6 +634,85 @@ public void setOwner(final Path path, final String owner,
final String group)
}
}
+ /**
+ * Set the value of an attribute for a path.
+ *
+ * @param path The path on which to set the attribute
+ * @param name The attribute to set
+ * @param value The byte value of the attribute to set (encoded in latin-1)
+ * @param flag The mode in which to set the attribute
+ * @throws IOException If there was an issue setting the attribute on Azure
+ * @throws IllegalArgumentException If name is null or empty or if value is
null
+ */
+ @Override
+ public void setXAttr(Path path, String name, byte[] value,
EnumSet flag)
+ throws IOException {
+LOG.debug(
+"AzureBlobFileSystem.setXAttr path: {}", path);
Review comment:
remove line break
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16660) ABFS: Make RetryCount in ExponentialRetryPolicy Configurable
[ https://issues.apache.org/jira/browse/HADOOP-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984015#comment-16984015 ] Hudson commented on HADOOP-16660: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17704 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17704/]) HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy (dazhou: rev 82ad9b549f9ae0867699042e0f5856577919a749) * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsRestOperationException.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ExponentialRetryPolicy.java > ABFS: Make RetryCount in ExponentialRetryPolicy Configurable > > > Key: HADOOP-16660 > URL: https://issues.apache.org/jira/browse/HADOOP-16660 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > ExponentialRetryPolicy will retry requests 30 times which is the default > retry count in code whe HttpRequests fail. > If the client too has re-try handling, the process seems hanged to the client > App due to the high number of retries. This Jira aims to provide a config > control for retry count. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16660) ABFS: Make RetryCount in ExponentialRetryPolicy Configurable
[ https://issues.apache.org/jira/browse/HADOOP-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16660: - Resolution: Fixed Status: Resolved (was: Patch Available) > ABFS: Make RetryCount in ExponentialRetryPolicy Configurable > > > Key: HADOOP-16660 > URL: https://issues.apache.org/jira/browse/HADOOP-16660 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > ExponentialRetryPolicy will retry requests 30 times which is the default > retry count in code whe HttpRequests fail. > If the client too has re-try handling, the process seems hanged to the client > App due to the high number of retries. This Jira aims to provide a config > control for retry count. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16660) ABFS: Make RetryCount in ExponentialRetryPolicy Configurable
[ https://issues.apache.org/jira/browse/HADOOP-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984000#comment-16984000 ] Da Zhou commented on HADOOP-16660: -- Commited. > ABFS: Make RetryCount in ExponentialRetryPolicy Configurable > > > Key: HADOOP-16660 > URL: https://issues.apache.org/jira/browse/HADOOP-16660 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > ExponentialRetryPolicy will retry requests 30 times which is the default > retry count in code whe HttpRequests fail. > If the client too has re-try handling, the process seems hanged to the client > App due to the high number of retries. This Jira aims to provide a config > control for retry count. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on issue #1697: Hadoop 16660: ABFS: Make RetryCount in ExponentialRetryPolicy Configurable
DadanielZ commented on issue #1697: Hadoop 16660: ABFS: Make RetryCount in ExponentialRetryPolicy Configurable URL: https://github.com/apache/hadoop/pull/1697#issuecomment-559281473 Committed, thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ closed pull request #1697: Hadoop 16660: ABFS: Make RetryCount in ExponentialRetryPolicy Configurable
DadanielZ closed pull request #1697: Hadoop 16660: ABFS: Make RetryCount in ExponentialRetryPolicy Configurable URL: https://github.com/apache/hadoop/pull/1697 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16455) ABFS: Implement FileSystem.access() method
[ https://issues.apache.org/jira/browse/HADOOP-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983886#comment-16983886 ] Virajith Jalaparti commented on HADOOP-16455: - Thanks for completing this [~bilahari.th]. Do you plan to backport to branch-2 as well? > ABFS: Implement FileSystem.access() method > -- > > Key: HADOOP-16455 > URL: https://issues.apache.org/jira/browse/HADOOP-16455 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 3.3.0 > > > Implement the access method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1707: HADOOP-16697. Tune/audit auth mode
hadoop-yetus commented on issue #1707: HADOOP-16697. Tune/audit auth mode URL: https://github.com/apache/hadoop/pull/1707#issuecomment-559224962 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 30m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 1m 7s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 18m 10s | trunk passed | | +1 :green_heart: | compile | 16m 55s | trunk passed | | +1 :green_heart: | checkstyle | 2m 36s | trunk passed | | +1 :green_heart: | mvnsite | 2m 15s | trunk passed | | +1 :green_heart: | shadedclient | 18m 41s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 12s | trunk passed | | +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 11s | trunk passed | | -0 :warning: | patch | 1m 32s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 20s | hadoop-aws in the patch failed. | | -1 :x: | compile | 15m 20s | root in the patch failed. | | -1 :x: | javac | 15m 20s | root in the patch failed. | | -0 :warning: | checkstyle | 2m 37s | root: The patch generated 15 new + 85 unchanged - 0 fixed = 100 total (was 85) | | -1 :x: | mvnsite | 0m 41s | hadoop-aws in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 12m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 13s | the patch passed | | -1 :x: | findbugs | 0m 39s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 9m 28s | hadoop-common in the patch failed. | | -1 :x: | unit | 0m 38s | hadoop-aws in the patch failed. | | +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. | | | | 146m 9s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ha.TestZKFailoverController | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1707 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ed9a770845eb 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9e69628 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-compile-root.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-compile-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/diff-checkstyle-root.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/testReport/ | | Max. process+thread count | 1420 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1707/10/console | | versions | git=2.7.4 maven=3.3.9
[GitHub] [hadoop] steveloughran commented on issue #1707: HADOOP-16697. Tune/audit auth mode
steveloughran commented on issue #1707: HADOOP-16697. Tune/audit auth mode URL: https://github.com/apache/hadoop/pull/1707#issuecomment-559176271 testing s3a ireland with & without auth. doing a local db test run too for completeness This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16455) ABFS: Implement FileSystem.access() method
[ https://issues.apache.org/jira/browse/HADOOP-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983670#comment-16983670 ] Hudson commented on HADOOP-16455: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17703 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17703/]) HADOOP-16455. ABFS: Implement FileSystem.access() method. (stevel: rev 9e69628f55be4acc040b7fe06cc22097238bb848) * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/constants/TestConfigurationKeys.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperationType.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCheckAccess.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java * (edit) hadoop-tools/hadoop-azure/src/site/markdown/abfs.md * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpQueryParams.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java > ABFS: Implement FileSystem.access() method > -- > > Key: HADOOP-16455 > URL: https://issues.apache.org/jira/browse/HADOOP-16455 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 3.3.0 > > > Implement the access method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1702: HDFS-14788 Use dynamic regex filter to ignore copy of source files in…
steveloughran commented on issue #1702: HDFS-14788 Use dynamic regex filter to ignore copy of source files in… URL: https://github.com/apache/hadoop/pull/1702#issuecomment-559150164 LGTM other than doc tuning This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1702: HDFS-14788 Use dynamic regex filter to ignore copy of source files in…
steveloughran commented on a change in pull request #1702: HDFS-14788 Use dynamic regex filter to ignore copy of source files in… URL: https://github.com/apache/hadoop/pull/1702#discussion_r351372167 ## File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm ## @@ -440,6 +440,23 @@ $H3 Copy-listing Generator of DistCp differs here from the legacy DistCp, in how paths are considered for copy. + One may also customize the filtering of files which shouldn't be copied + by passing the current supported implementation of CopyFilter interface + or a new implementation can be written. This can be specified by setting the + `distcp.filters.class` in the DistCpOptions: + + 1. `distcp.filters.class` to RegexCopyFilter. If you are using this implementation, + you will have to pass along CopyFilter`distcp.filters.file` which contains the regex Review comment: 1. add space before the tick 2. use `quotes` around all classnames This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1702: HDFS-14788 Use dynamic regex filter to ignore copy of source files in…
hadoop-yetus removed a comment on issue #1702: HDFS-14788 Use dynamic regex filter to ignore copy of source files in… URL: https://github.com/apache/hadoop/pull/1702#issuecomment-554912253 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1809 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1076 | trunk passed | | +1 | compile | 29 | trunk passed | | +1 | checkstyle | 23 | trunk passed | | +1 | mvnsite | 31 | trunk passed | | +1 | shadedclient | 799 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 24 | trunk passed | | 0 | spotbugs | 44 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 42 | trunk passed | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 17 | hadoop-distcp in the patch failed. | | -1 | compile | 16 | hadoop-distcp in the patch failed. | | -1 | javac | 16 | hadoop-distcp in the patch failed. | | -0 | checkstyle | 16 | hadoop-tools/hadoop-distcp: The patch generated 39 new + 21 unchanged - 0 fixed = 60 total (was 21) | | -1 | mvnsite | 18 | hadoop-distcp in the patch failed. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 795 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 21 | the patch passed | | -1 | findbugs | 20 | hadoop-distcp in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 20 | hadoop-distcp in the patch failed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 4889 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1702 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f806b6a49007 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 96c4520 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/patch-mvninstall-hadoop-tools_hadoop-distcp.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/patch-compile-hadoop-tools_hadoop-distcp.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/patch-compile-hadoop-tools_hadoop-distcp.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/patch-mvnsite-hadoop-tools_hadoop-distcp.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/patch-findbugs-hadoop-tools_hadoop-distcp.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/testReport/ | | Max. process+thread count | 421 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1702/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16455) ABFS: Implement FileSystem.access() method
[ https://issues.apache.org/jira/browse/HADOOP-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16455. - Fix Version/s: 3.3.0 Resolution: Fixed > ABFS: Implement FileSystem.access() method > -- > > Key: HADOOP-16455 > URL: https://issues.apache.org/jira/browse/HADOOP-16455 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 3.3.0 > > > Implement the access method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1711: HADOOP-16455. ABFS: Implement FileSystem.access() method.
steveloughran commented on issue #1711: HADOOP-16455. ABFS: Implement FileSystem.access() method. URL: https://github.com/apache/hadoop/pull/1711#issuecomment-559146956 +1, merged via github. If you want on branch-3.2, cherry pick, retest and provide a new PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1711: HADOOP-16455. ABFS: Implement FileSystem.access() method.
steveloughran merged pull request #1711: HADOOP-16455. ABFS: Implement FileSystem.access() method. URL: https://github.com/apache/hadoop/pull/1711 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1711: HADOOP-16455. ABFS: Implement FileSystem.access() method.
hadoop-yetus removed a comment on issue #1711: HADOOP-16455. ABFS: Implement FileSystem.access() method. URL: https://github.com/apache/hadoop/pull/1711#issuecomment-555448798 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 41s | trunk passed | | +1 :green_heart: | compile | 0m 30s | trunk passed | | +1 :green_heart: | checkstyle | 0m 21s | trunk passed | | +1 :green_heart: | mvnsite | 0m 31s | trunk passed | | +1 :green_heart: | shadedclient | 13m 0s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed | | +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 49s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 26s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 4 new + 6 unchanged - 1 fixed = 10 total (was 7) | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 29s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | | +1 :green_heart: | findbugs | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 21s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 54m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1711/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1711 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux dbcb32b22aae 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 3cecb2a | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1711/7/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1711/7/testReport/ | | Max. process+thread count | 429 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1711/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1712: HADOOP-16699: Adding verbose TRACE logging
steveloughran commented on issue #1712: HADOOP-16699: Adding verbose TRACE
logging
URL: https://github.com/apache/hadoop/pull/1712#issuecomment-559143920
If you are actually trying to debug the behaviour of plug-in components I
would prefer that you embrace `org.apache.hadoop.util.DurationInfo` which can
you log out in fellow debug at the start and end of all operations, and
calculates actual durations -which is handy
example use
```java
try (DurationInfo d =
new DurationInfo(LOG, false, "Waiting for task completion")) {
future.join();
} catch (CompletionException e) {
raiseInnerCause(e);
}
```
No matter the code flow, information will get logged. With your current
patch if there is a failure nobody sees it.
You've been working here, if you're happy with it: fine. However, you can do
more and for all your future patches I will be expecting it.
So: let me know what you want to do here?
1. Merge in as is
2. Explore DurationInfo
Your choice.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1712: HADOOP-16699: Adding verbose TRACE logging
steveloughran commented on a change in pull request #1712: HADOOP-16699: Adding
verbose TRACE logging
URL: https://github.com/apache/hadoop/pull/1712#discussion_r351358065
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -143,8 +143,12 @@ public AzureBlobFileSystemStore(URI uri, boolean
isSecureScheme, Configuration c
} catch (IllegalAccessException exception) {
throw new FileSystemOperationUnhandledException(exception);
}
+
+LOG.trace("AbfsConfiguration init complete");
+
this.userGroupInformation = UserGroupInformation.getCurrentUser();
this.userName = userGroupInformation.getShortUserName();
+LOG.trace("UGI init complete");
Review comment:
well, if its in, print the username
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1712: HADOOP-16699: Adding verbose TRACE logging
hadoop-yetus removed a comment on issue #1712: HADOOP-16699: Adding verbose TRACE logging URL: https://github.com/apache/hadoop/pull/1712#issuecomment-553285423 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 3109 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1085 | trunk passed | | +1 | compile | 33 | trunk passed | | +1 | checkstyle | 26 | trunk passed | | +1 | mvnsite | 35 | trunk passed | | +1 | shadedclient | 823 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 28 | trunk passed | | 0 | spotbugs | 51 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 50 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 29 | the patch passed | | +1 | compile | 25 | the patch passed | | +1 | javac | 25 | the patch passed | | -0 | checkstyle | 18 | hadoop-tools/hadoop-azure: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) | | +1 | mvnsite | 28 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 816 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | +1 | findbugs | 54 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 83 | hadoop-azure in the patch passed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 6401 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1712/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1712 | | JIRA Issue | HADOOP-16699 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4572337f1d45 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / df6b316 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1712/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1712/1/testReport/ | | Max. process+thread count | 454 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1712/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1732: HADOOP-16729. Extract version numbers to head of pom.xml
hadoop-yetus commented on issue #1732: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1732#issuecomment-559123362 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 945 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-3.2 Compile Tests _ | | +1 | mvninstall | 1273 | branch-3.2 passed | | +1 | compile | 14 | branch-3.2 passed | | +1 | mvnsite | 17 | branch-3.2 passed | | +1 | shadedclient | 2098 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 18 | branch-3.2 passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 12 | the patch passed | | +1 | compile | 10 | the patch passed | | +1 | javac | 10 | the patch passed | | +1 | mvnsite | 12 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 858 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 13 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 12 | hadoop-project in the patch passed. | | -1 | asflicense | 26 | The patch generated 1 ASF License warnings. | | | | 4147 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1732/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1732 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux 0a718b750afb 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | branch-3.2 / 66e58b5 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1732/1/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1732/1/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 305 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1732/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
hadoop-yetus commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-559117632 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 31m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 20m 46s | trunk passed | | +1 :green_heart: | compile | 0m 15s | trunk passed | | +1 :green_heart: | mvnsite | 0m 18s | trunk passed | | +1 :green_heart: | shadedclient | 35m 22s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 17s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 11s | the patch passed | | +1 :green_heart: | compile | 0m 11s | the patch passed | | +1 :green_heart: | javac | 0m 11s | the patch passed | | +1 :green_heart: | mvnsite | 0m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 2s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 13s | hadoop-project in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 85m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1730/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1730 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux c676926014b6 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7f2ea2a | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1730/2/testReport/ | | Max. process+thread count | 293 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1730/2/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16731) 编译hadoop3.2.1源码出错
zhaobaoquan created HADOOP-16731: Summary: 编译hadoop3.2.1源码出错 Key: HADOOP-16731 URL: https://issues.apache.org/jira/browse/HADOOP-16731 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.2.1 Environment: ubuntu16.04 32位 hadoop 3.2.1 Reporter: zhaobaoquan Fix For: 3.2.1 Attachments: lll.log [WARNING] make[1]: *** [CMakeFiles/nativetask_static.dir/all] Error 2 [WARNING] make: *** [all] Error 2 [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.1:cmake-compile (cmake-compile) on project hadoop-mapreduce-client-nativetask: make failed with error code 2 -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.1:cmake-compile (cmake-compile) on project hadoop-mapreduce-client-nativetask: make failed with error code 2 at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with error code 2 at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake (CompileMojo.java:229) at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute (CompileMojo.java:98) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.or
[GitHub] [hadoop] tamaashu commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
tamaashu commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-559087752 PR for branch 3.2: https://github.com/apache/hadoop/pull/1732 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tamaashu commented on issue #1732: HADOOP-16729. Extract version numbers to head of pom.xml
tamaashu commented on issue #1732: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1732#issuecomment-559087895 PR for trunk: https://github.com/apache/hadoop/pull/1730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tamaashu opened a new pull request #1732: HADOOP-16729. Extract version numbers to head of pom.xml
tamaashu opened a new pull request #1732: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1732 For branch-3.2 To be able to easily replace third-party dependency version numbers it would be useful to collect their version numbers at the top of the pom.xml. For many third-parties (e.g. slf4j, jetty, etc.) this is already done, but I would need the same for others. The change doesn't have any effect on the code or the build, no version numbers would be changed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tamaashu commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
tamaashu commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-559081935 @adamantal done > It might make sense to put the log4j version right after the slf4j one a bit above, but it's perfectly okay in this way as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16729) Extract version numbers to head of pom.xml
[ https://issues.apache.org/jira/browse/HADOOP-16729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983473#comment-16983473 ] Adam Antal commented on HADOOP-16729: - Maybe this could be a subtask of HADOOP-9991. > Extract version numbers to head of pom.xml > -- > > Key: HADOOP-16729 > URL: https://issues.apache.org/jira/browse/HADOOP-16729 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Tamas Penzes >Assignee: Tamas Penzes >Priority: Minor > > To be able to easily replace third-party dependency version numbers it would > be useful to collect their version numbers at the top of the pom.xml. > For many third-parties (e.g. slf4j, jetty, etc.) this is already done, but I > would need the same for others. The change doesn't have any effect on the > code or the build, no version numbers would be changed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adamantal commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
adamantal commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-559065447 Thank you for the patch @tamaashu! +1 (non-binding) It might make sense to put the log4j version right after the slf4j one a bit above, but it's perfectly okay in this way as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1731: Initial draft for TAS support
hadoop-yetus commented on issue #1731: Initial draft for TAS support URL: https://github.com/apache/hadoop/pull/1731#issuecomment-559046906 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 21m 44s | trunk passed | | +1 :green_heart: | compile | 0m 29s | trunk passed | | +1 :green_heart: | checkstyle | 0m 23s | trunk passed | | +1 :green_heart: | mvnsite | 0m 33s | trunk passed | | +1 :green_heart: | shadedclient | 15m 2s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed | | +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 57s | trunk passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 23s | hadoop-azure in the patch failed. | | -1 :x: | compile | 0m 24s | hadoop-azure in the patch failed. | | -1 :x: | javac | 0m 24s | hadoop-azure in the patch failed. | | -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 25 new + 5 unchanged - 0 fixed = 30 total (was 5) | | -1 :x: | mvnsite | 0m 23s | hadoop-azure in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 56s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 19s | hadoop-tools_hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | -1 :x: | findbugs | 0m 23s | hadoop-azure in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 24s | hadoop-azure in the patch failed. | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 60m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1731 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 606028123db8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7f2ea2a | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/testReport/ | | Max. process+thread count | 308 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1731/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Ap
[GitHub] [hadoop] tamaashu commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
tamaashu commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-559042604 There is no need for additional or modified tests since, I didn't change any code. The only change is that I define third parties' version numbers in a different place. (Didn't even change version numbers.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya opened a new pull request #1731: Initial draft for TAS support
snvijaya opened a new pull request #1731: Initial draft for TAS support URL: https://github.com/apache/hadoop/pull/1731 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16728) SFTPFileSystem connection pool not reusing connections properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated HADOOP-16728: Labels: SFTP fs perfomance (was: SFTP) > SFTPFileSystem connection pool not reusing connections properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > Labels: SFTP, fs, perfomance > > This issue is trying to address few issues in the SFTP filesystem, and > especially in SFTPConnectionPool: > * all channels for a given connectionInfo are removed from the pool instead > of only the selected channel. > * Closed connections are added back into the pool. They should be filtered > out. > * The SFTPInputStream is closing the stream instead of returning it into the > pool. > * Some connection leak could happen in some cases > As a result, if you have multiple threads sharing the pool, the SFTP > filesystem is underperforming by untimely re-establishing SSH connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16728) SFTPFileSystem connection pool not reusing connections properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated HADOOP-16728: Labels: SFTP (was: ) > SFTPFileSystem connection pool not reusing connections properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > Labels: SFTP > > This issue is trying to address few issues in the SFTP filesystem, and > especially in SFTPConnectionPool: > * all channels for a given connectionInfo are removed from the pool instead > of only the selected channel. > * Closed connections are added back into the pool. They should be filtered > out. > * The SFTPInputStream is closing the stream instead of returning it into the > pool. > * Some connection leak could happen in some cases > As a result, if you have multiple threads sharing the pool, the SFTP > filesystem is underperforming by untimely re-establishing SSH connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16728) SFTPFileSystem connection pool not reusing connections properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated HADOOP-16728: Summary: SFTPFileSystem connection pool not reusing connections properly (was: SFTPFileSystem connection pooling is not handling the pool properly) > SFTPFileSystem connection pool not reusing connections properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > > This issue is trying to address few issues in the SFTP filesystem, and > especially in SFTPConnectionPool: > * all channels for a given connectionInfo are removed from the pool instead > of only the selected channel. > * Closed connections are added back into the pool. They should be filtered > out. > * The SFTPInputStream is closing the stream instead of returning it into the > pool. > * Some connection leak could happen in some cases > As a result, if you have multiple threads sharing the pool, the SFTP > filesystem is underperforming by untimely re-establishing SSH connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16728) SFTPFileSystem connection pooling is not handling the pool properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983256#comment-16983256 ] Benoit Perroud commented on HADOOP-16728: - While digging into this issue, I found 2 more problems: * Closed connections are added back into the pool. They should be filtered out. * The SFTPInputStream is closing the stream instead of returning it into the pool. I'll provide an update of the PR to fix these two issues, too. > SFTPFileSystem connection pooling is not handling the pool properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > > This issue is trying to address few issues in the SFTP filesystem, and > especially in SFTPConnectionPool: > * all channels for a given connectionInfo are removed from the pool instead > of only the selected channel. > * Closed connections are added back into the pool. They should be filtered > out. > * The SFTPInputStream is closing the stream instead of returning it into the > pool. > * Some connection leak could happen in some cases > As a result, if you have multiple threads sharing the pool, the SFTP > filesystem is underperforming by untimely re-establishing SSH connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16728) SFTPFileSystem connection pooling is not handling the pool properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated HADOOP-16728: Description: This issue is trying to address few issues in the SFTP filesystem, and especially in SFTPConnectionPool: * all channels for a given connectionInfo are removed from the pool instead of only the selected channel. * Closed connections are added back into the pool. They should be filtered out. * The SFTPInputStream is closing the stream instead of returning it into the pool. * Some connection leak could happen in some cases As a result, if you have multiple threads sharing the pool, the SFTP filesystem is underperforming by untimely re-establishing SSH connections. was: This issue is trying to address few issues in the SFTP filesystem, and especially in SFTPConnectionPool: * all channels for a given connectionInfo are removed from the pool instead of only the selected channel. * As a result, if you have 2 threads sharing the pool, the first one acquiring a channels removes all the others, leading to inefficient SSH reconnections. > SFTPFileSystem connection pooling is not handling the pool properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > > This issue is trying to address few issues in the SFTP filesystem, and > especially in SFTPConnectionPool: > * all channels for a given connectionInfo are removed from the pool instead > of only the selected channel. > * Closed connections are added back into the pool. They should be filtered > out. > * The SFTPInputStream is closing the stream instead of returning it into the > pool. > * Some connection leak could happen in some cases > As a result, if you have multiple threads sharing the pool, the SFTP > filesystem is underperforming by untimely re-establishing SSH connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
