[jira] [Commented] (HADOOP-16739) Fix native build failure of hadoop-pipes on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-16739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033391#comment-17033391 ] Akira Ajisaka commented on HADOOP-16739: Thank you, [~iwasakims]! > Fix native build failure of hadoop-pipes on CentOS 8 > > > Key: HADOOP-16739 > URL: https://issues.apache.org/jira/browse/HADOOP-16739 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/pipes >Affects Versions: 2.10.0, 3.2.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16739.001.patch > > > Native build fails in hadoop-tools/hadoop-pips on CentOS 8 due to lack of > rpc.h which was removed from glibc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16739) Fix native build failure of hadoop-pipes on CentOS 8
[
https://issues.apache.org/jira/browse/HADOOP-16739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HADOOP-16739:
--
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed this to trunk. The contents added to BUILDING.txt is only
applidcable to trunk (protobuf-3.7.1).
If you want to build branch-2.10 with {{-Pnative}} on CentOS 8, you need the
fix of HADOOP-14597 too. The patches of HADOOP-16739 and HADOOP-14597 are
cleanly applicable on today's branch-2.10.
> Fix native build failure of hadoop-pipes on CentOS 8
>
>
> Key: HADOOP-16739
> URL: https://issues.apache.org/jira/browse/HADOOP-16739
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/pipes
>Affects Versions: 2.10.0, 3.2.1
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16739.001.patch
>
>
> Native build fails in hadoop-tools/hadoop-pips on CentOS 8 due to lack of
> rpc.h which was removed from glibc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13951) Precommit builds do not adequately protect against test malformed fs permissions.
[
https://issues.apache.org/jira/browse/HADOOP-13951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033384#comment-17033384
]
Akira Ajisaka commented on HADOOP-13951:
I noticed the cleanup job only cleans "Precommit-HDFS-build". Added a parameter
to specify the job name to clean up other Jenkins jobs.
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-clean-hdfs-workspace/16/console
> Precommit builds do not adequately protect against test malformed fs
> permissions.
> -
>
> Key: HADOOP-13951
> URL: https://issues.apache.org/jira/browse/HADOOP-13951
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Sean Busbey
>Priority: Critical
>
> Right now this is expressed as failed Precommit-YARN-build jobs when they run
> on H5 / H6 (see INFRA-13148), but the problem exists for all of the hadoop
> related precommit jobs.
> The issue is that we have some tests in Common (and maybe HDFS) that
> purposefully set permissions within the {{target/}} directory to simulate a
> failure to interact with underlying fs data. The test sets some
> subdirectories to have permissions such that we can no longer delete their
> contents.
> Right now our precommit jobs include a step post-yetus-test-patch that
> traverses the target directories and ensures that all subdirectories are
> modifiable:
> {code}
> find ${WORKSPACE} -name target | xargs chmod -R u+w
> {code}
> Unfortunately, if we don't get to that line (say due to an aborted build, or
> if the call to yetus test-patch exceeds the job timeout), then we are left in
> a state where there are still subdirectories that can't be modified
> (including deleted).
> Our builds also currently attempt to run a {{git clean}} at the very start of
> the build after the repo is updated. If we have one of the aforementioned
> timeouts that leaves a can't-be-deleted test directory, then all future
> builds on that machine will fail attempting to run the {{git clean}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13951) Precommit builds do not adequately protect against test malformed fs permissions.
[
https://issues.apache.org/jira/browse/HADOOP-13951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033378#comment-17033378
]
Akira Ajisaka commented on HADOOP-13951:
H17 is failing
([https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1406/console)|https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1406/console]
and ran
[https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-clean-hdfs-workspace/]
on the node.
Thanks Sean and Andrew for the work.
> Precommit builds do not adequately protect against test malformed fs
> permissions.
> -
>
> Key: HADOOP-13951
> URL: https://issues.apache.org/jira/browse/HADOOP-13951
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Sean Busbey
>Priority: Critical
>
> Right now this is expressed as failed Precommit-YARN-build jobs when they run
> on H5 / H6 (see INFRA-13148), but the problem exists for all of the hadoop
> related precommit jobs.
> The issue is that we have some tests in Common (and maybe HDFS) that
> purposefully set permissions within the {{target/}} directory to simulate a
> failure to interact with underlying fs data. The test sets some
> subdirectories to have permissions such that we can no longer delete their
> contents.
> Right now our precommit jobs include a step post-yetus-test-patch that
> traverses the target directories and ensures that all subdirectories are
> modifiable:
> {code}
> find ${WORKSPACE} -name target | xargs chmod -R u+w
> {code}
> Unfortunately, if we don't get to that line (say due to an aborted build, or
> if the call to yetus test-patch exceeds the job timeout), then we are left in
> a state where there are still subdirectories that can't be modified
> (including deleted).
> Our builds also currently attempt to run a {{git clean}} at the very start of
> the build after the repo is updated. If we have one of the aforementioned
> timeouts that leaves a can't-be-deleted test directory, then all future
> builds on that machine will fail attempting to run the {{git clean}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16739) Fix native build failure of hadoop-pipes on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-16739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033370#comment-17033370 ] Hudson commented on HADOOP-16739: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17936 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17936/]) HADOOP-16739. Fix native build failure of hadoop-pipes on CentOS 8. (iwasakims: rev d5467d299db3dddcbcd2f77a281d5fa82c4a9e4b) * (edit) BUILDING.txt * (edit) hadoop-tools/hadoop-pipes/src/CMakeLists.txt > Fix native build failure of hadoop-pipes on CentOS 8 > > > Key: HADOOP-16739 > URL: https://issues.apache.org/jira/browse/HADOOP-16739 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/pipes >Affects Versions: 2.10.0, 3.2.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HADOOP-16739.001.patch > > > Native build fails in hadoop-tools/hadoop-pips on CentOS 8 due to lack of > rpc.h which was removed from glibc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
aajisaka commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router URL: https://github.com/apache/hadoop/pull/1832#issuecomment-583946905 @goiri Thank you for your review. Fixed checkstyle issues. > Yetus does not look very happy but it seems unrelated. Agreed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16739) Fix native build failure of hadoop-pipes on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-16739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033361#comment-17033361 ] Masatake Iwasaki commented on HADOOP-16739: --- Thanks [~aajisaka]. I'm committing this. > Fix native build failure of hadoop-pipes on CentOS 8 > > > Key: HADOOP-16739 > URL: https://issues.apache.org/jira/browse/HADOOP-16739 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/pipes >Affects Versions: 2.10.0, 3.2.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HADOOP-16739.001.patch > > > Native build fails in hadoop-tools/hadoop-pips on CentOS 8 due to lack of > rpc.h which was removed from glibc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16739) Fix native build failure of hadoop-pipes on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-16739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1703#comment-1703 ] Akira Ajisaka commented on HADOOP-16739: +1, I could compile hadoop-pipes successfully with your patch. > Fix native build failure of hadoop-pipes on CentOS 8 > > > Key: HADOOP-16739 > URL: https://issues.apache.org/jira/browse/HADOOP-16739 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/pipes >Affects Versions: 2.10.0, 3.2.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HADOOP-16739.001.patch > > > Native build fails in hadoop-tools/hadoop-pips on CentOS 8 due to lack of > rpc.h which was removed from glibc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16739) Fix native build failure of hadoop-pipes on CentOS 8
[ https://issues.apache.org/jira/browse/HADOOP-16739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-16739: -- Affects Version/s: 2.10.0 3.2.1 > Fix native build failure of hadoop-pipes on CentOS 8 > > > Key: HADOOP-16739 > URL: https://issues.apache.org/jira/browse/HADOOP-16739 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/pipes >Affects Versions: 2.10.0, 3.2.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HADOOP-16739.001.patch > > > Native build fails in hadoop-tools/hadoop-pips on CentOS 8 due to lack of > rpc.h which was removed from glibc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1839: HADOOP-16848. Refactoring: initial layering
hadoop-yetus commented on issue #1839: HADOOP-16848. Refactoring: initial layering URL: https://github.com/apache/hadoop/pull/1839#issuecomment-583900610 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 0m 26s | root in trunk failed. | | -1 :x: | compile | 0m 28s | root in trunk failed. | | -0 :warning: | checkstyle | 2m 58s | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 0m 59s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 10m 39s | branch has errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 31s | trunk passed | | +0 :ok: | spotbugs | 1m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 6s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 54s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 29s | hadoop-aws in the patch failed. | | -1 :x: | compile | 16m 23s | root in the patch failed. | | -1 :x: | javac | 16m 23s | root in the patch failed. | | -0 :warning: | checkstyle | 2m 37s | root: The patch generated 36 new + 0 unchanged - 0 fixed = 36 total (was 0) | | -1 :x: | mvnsite | 0m 47s | hadoop-aws in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 19s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 43s | the patch passed | | -1 :x: | findbugs | 0m 47s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 9m 37s | hadoop-common in the patch passed. | | -1 :x: | unit | 0m 47s | hadoop-aws in the patch failed. | | +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | | | | 78m 12s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.viewfs.TestViewFsTrash | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1839 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e7211e3ac278 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6191d4b | | Default Java | 1.8.0_242 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/branch-mvninstall-root.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/branch-compile-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1839/out/maven-branch-checkstyle-root.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-compile-root.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-compile-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/diff-checkstyle-root.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit |
[GitHub] [hadoop] steveloughran commented on issue #1839: HADOOP-16848. Refactoring: initial layering
steveloughran commented on issue #1839: HADOOP-16848. Refactoring: initial layering URL: https://github.com/apache/hadoop/pull/1839#issuecomment-583892914 FYI @bgaborg @sidseth not quite ready for review, but you get the idea. Split up the S3AFileSystem class into layers with interface and impl, async chained init from S3A FS, which creates them, knows about DelegationTokens, and how to bind its operations to the store This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1839: HADOOP-16848. Refactoring: initial layering
steveloughran opened a new pull request #1839: HADOOP-16848. Refactoring: initial layering URL: https://github.com/apache/hadoop/pull/1839 First PoC of My planned layout model of the S3A FS. * There's a raw layer and a guarded layer * which are instantiated in sequence in a separate executor from S3AFileSystem.initalize And whose accessors block until completed or rethrow failures. The layers are being handed in all their dependencies from FS.initialize() and we currently block until started. What I plan to do, in a future iteration, is: * each layer extracts their own settings from the config and stores locally (list version, upload size etc) * have each layer instantiate their internal classes (AWS S3 client, transfer manager) internally * Also async create: metastore, DT binding * And all startup actions (check bucket, init multipart, ...) Then * move ops to the layers, raw* -> rawStore; inner -> S3AStore * move WriteOperationHelper, SelectBinding, etc, to all work against S3AStore rather than FS. S3AStore will become where most of the code moves to; S3AFilesystem more of the init and binding to hadoop FS API. RawS3A will be the accessor through which all AWS client access goes. Not going to change: all accessors on S3AFileSystem...not just tests use it but some external code (cloudstore) needs it to get at low level S3A, etc. Change-Id: I998c0d61cce2ee7fd0be804bf21da6b68fd69a6f ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16848) add initial S3A layering + async init
[ https://issues.apache.org/jira/browse/HADOOP-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033292#comment-17033292 ] Gabor Bota commented on HADOOP-16848: - Good idea, happy to do the review and help with the work. > add initial S3A layering + async init > - > > Key: HADOOP-16848 > URL: https://issues.apache.org/jira/browse/HADOOP-16848 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Split the S3A code into layers > * S3AFileSystem > * S3AStore + Impl > * RawS3A + Impl > S3AFS will create the others and start in order: DelegationTokens, RawS3A, > S3AStore, Metastore > this will involve wrapping all access of DTs, s3client, Metastore to block > until that layer is complete, or raise an exception if instantiation of > it/predecessor failed. > New layers will all be subclasses of Service, split into Interface and Impl, > so we can manage the init/start/stop lifecycle with existing code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13447) Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
[
https://issues.apache.org/jira/browse/HADOOP-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13447:
Parent: (was: HADOOP-16456)
Issue Type: Improvement (was: Sub-task)
> Refactor S3AFileSystem to support introduction of separate metadata
> repository and tests.
> -
>
> Key: HADOOP-13447
> URL: https://issues.apache.org/jira/browse/HADOOP-13447
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13447-HADOOP-13446.001.patch,
> HADOOP-13447-HADOOP-13446.002.patch, HADOOP-13447.003.patch,
> HADOOP-13447.004.patch, HADOOP-13447.005.patch
>
>
> The scope of this issue is to refactor the existing {{S3AFileSystem}} into
> multiple coordinating classes. The goal of this refactoring is to separate
> the {{FileSystem}} API binding from the AWS SDK integration, make code
> maintenance easier while we're making changes for S3Guard, and make it easier
> to mock some implementation details so that tests can simulate eventual
> consistency behavior in a deterministic way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16830) Add public IOStatistics API; S3A to collect and report across threads
[ https://issues.apache.org/jira/browse/HADOOP-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16830: Parent Issue: HADOOP-16456 (was: HADOOP-16829) > Add public IOStatistics API; S3A to collect and report across threads > - > > Key: HADOOP-16830 > URL: https://issues.apache.org/jira/browse/HADOOP-16830 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Applications like to collect the statistics which specific operations take, > by collecting exactly those operations done during the execution of FS API > calls by their individual worker threads, and returning these to their job > driver > * S3A has a statistics API for some streams, but it's a non-standard one; > Impala can't use it > * FileSystem storage statistics are public, but as they aren't cross-thread, > they don't aggregate properly > Proposed > # A new IOStatistics interface to serve up statistics > # S3A to implement > # other stores to follow > # Pass-through from the usual wrapper classes (FS data input/output streams) > It's hard to think about how best to offer an API for operation context > stats, and how to actually implement. > ThreadLocal isn't enough because the helper threads need to update on the > thread local value of the instigator > My Initial PoC doesn't address that issue, but it shows what I'm thinking of -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16848) add initial S3A layering + async init
Steve Loughran created HADOOP-16848: --- Summary: add initial S3A layering + async init Key: HADOOP-16848 URL: https://issues.apache.org/jira/browse/HADOOP-16848 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Reporter: Steve Loughran Assignee: Steve Loughran Split the S3A code into layers * S3AFileSystem * S3AStore + Impl * RawS3A + Impl S3AFS will create the others and start in order: DelegationTokens, RawS3A, S3AStore, Metastore this will involve wrapping all access of DTs, s3client, Metastore to block until that layer is complete, or raise an exception if instantiation of it/predecessor failed. New layers will all be subclasses of Service, split into Interface and Impl, so we can manage the init/start/stop lifecycle with existing code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13447) Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
[
https://issues.apache.org/jira/browse/HADOOP-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13447:
Parent Issue: HADOOP-16456 (was: HADOOP-13345)
> Refactor S3AFileSystem to support introduction of separate metadata
> repository and tests.
> -
>
> Key: HADOOP-13447
> URL: https://issues.apache.org/jira/browse/HADOOP-13447
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13447-HADOOP-13446.001.patch,
> HADOOP-13447-HADOOP-13446.002.patch, HADOOP-13447.003.patch,
> HADOOP-13447.004.patch, HADOOP-13447.005.patch
>
>
> The scope of this issue is to refactor the existing {{S3AFileSystem}} into
> multiple coordinating classes. The goal of this refactoring is to separate
> the {{FileSystem}} API binding from the AWS SDK integration, make code
> maintenance easier while we're making changes for S3Guard, and make it easier
> to mock some implementation details so that tests can simulate eventual
> consistency behavior in a deterministic way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16838) Support for `fs.s3a.endpoint.region`
[ https://issues.apache.org/jira/browse/HADOOP-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033237#comment-17033237 ] Steve Loughran commented on HADOOP-16838: - oh, and if you are wondering why we expect hostnames for endpoint and not region names is (a) to support third party stores like minio and (b) so when AWS add a new region there's no need to update the SDKs on every shipping hadoop version, retest, patch where needed and then rerelease > Support for `fs.s3a.endpoint.region` > > > Key: HADOOP-16838 > URL: https://issues.apache.org/jira/browse/HADOOP-16838 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Nitish >Priority: Major > > Currently it is not possible to connect S3 Compatible services like MinIO, > Ceph, etc (running with a custom region) to Spark with s3a connector. For > example, if MinIO is running on a Server with > * IP Address: 192.168.0.100 > * Region: ap-southeast-1 > The s3a connector can't be configured to use the region `ap-southeast-1`. > It would be great to have a configuration field like > `fs.s3a.endpoint.region`. This will be very helpful for users deploying > Private Cloud and who intend to use S3 like services on premises. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
hadoop-yetus commented on issue #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init() URL: https://github.com/apache/hadoop/pull/1838#issuecomment-583830785 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 27m 6s | trunk passed | | +1 :green_heart: | compile | 0m 38s | trunk passed | | +1 :green_heart: | checkstyle | 0m 33s | trunk passed | | +1 :green_heart: | mvnsite | 0m 47s | trunk passed | | +1 :green_heart: | shadedclient | 20m 8s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 35s | trunk passed | | +0 :ok: | spotbugs | 1m 22s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 15s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 39s | the patch passed | | +1 :green_heart: | compile | 0m 34s | the patch passed | | +1 :green_heart: | javac | 0m 34s | the patch passed | | -0 :warning: | checkstyle | 0m 26s | hadoop-tools/hadoop-aws: The patch generated 22 new + 15 unchanged - 0 fixed = 37 total (was 15) | | +1 :green_heart: | mvnsite | 0m 42s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 17m 52s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 29s | the patch passed | | -1 :x: | findbugs | 1m 22s | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 78m 33s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-tools/hadoop-aws | | | Switch statement found in org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing() where default case is missing At S3AFileSystem.java:where default case is missing At S3AFileSystem.java:[lines 463-470] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1838 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux d8622c0ac2b8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d23317b | | Default Java | 1.8.0_232 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/whitespace-eol.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/testReport/ | | Max. process+thread count | 342 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1838/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail:
[GitHub] [hadoop] mukund-thakur opened a new pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init()
mukund-thakur opened a new pull request #1838: HADOOP-16711 Add way to skip verifyBuckets check in S3A fs init() URL: https://github.com/apache/hadoop/pull/1838 Testing Bucket used : https://mthakur-data.s3.ap-south-1.amazonaws.com/file2 Ran all the UT's and IT's using default settings with S3guard. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
