[jira] [Created] (HADOOP-16454) Share delegation tokens between multiple HttpFS servers
Akira Ajisaka created HADOOP-16454: -- Summary: Share delegation tokens between multiple HttpFS servers Key: HADOOP-16454 URL: https://issues.apache.org/jira/browse/HADOOP-16454 Project: Hadoop Common Issue Type: New Feature Components: httpfs Environment: Kerberized, clients connect to multiple HttpFS servers via load balancer Reporter: Akira Ajisaka In our environment, multiple HttpFS servers are deployed for the clients outside the HDFS cluster. As we are using external load balancer service for the HttpFS servers, the following situation may happen: 1. A client authenticates with a HttpFS server and gets a delegation token. Using the delegation token, the client can access to the NameNode. 2. In the next session, the client authenticates with another HttpFS server (via load balancer) using the same delegation token. The client fails to access because the other HttpFS server does not have the information of the delegation token. It would be nice to have a feature like HADOOP-14445 in HttpFS to fix the situation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Aug Hadoop Community Meetup in China
Hi Junping My team and I work for helping more open source projects run on ARM server, foucs on big data projects in this stage, like: Hadoop, Spark, Flink and so on, I would like to share a topic about it with local community, include: current progress, faced issue and future plan. Thank you holding the meetup and glad to see Apache folks in the meetup :) RuiChen https://openlabtesting.org/ https://github.com/theopenlab Weiwei Yang 于2019年7月23日周二 下午5:43写道： > Hi Junping > > Thanks. I would like to get a slot to talk about our new open source > project: YuniKorn. > > Thanks > Weiwei > On Jul 23, 2019, 5:08 PM +0800, 俊平堵 , wrote: > > Thanks for these positive feedbacks! The local community has voted the > date and location to be 8/10, Beijing. So please book your time ahead if > you have interest to join. > > I have gathered a few topics, and some candidate places for hosting this > meetup. If you would like to propose more topics, please nominate it here > or ping me before this weekend (7/28, CST time). > > Will update here when I have more to share. thx! > > > > > > > > > > <> > > > > <> > > > > > > > > Thanks, > > > > Junping > > > > > 俊平堵 于2019年7月18日周四 下午3:28写道： > > > > Hi, all! > > > > I am glad to let you know that we are organizing > Hadoop Contributors Meetup in China on Aug. > > > > > > > > This could be the first time hadoop community meetup in China and > many attendees are expected to come from big data pioneers, such as: > Cloudera, Tencent, Alibaba, Xiaomi, Didi, JD, Meituan, Toutiao, Sina, etc. > > > > > > > > We're still working out the details, such as dates, contents and > locations. Here is a quick survey: https://www.surveymonkey.com/r/Y99RT3W > where you can vote your prefer dates and locations if you would like to > attend - the survey will end in July. 21. 12PM China Standard Time, and > result will go public in next day. > > > > > > > > Also, please feel free to reach out to me if you have a topic to > propose for the meetup. Will send out an update later with more details > when I get more to share. Thanks! > > > > > > > > Cheers, > > > > > > > > Junping >
[jira] [Created] (HADOOP-16452) Increase ipc.maximum.data.length default from 64MB to 128MB
Wei-Chiu Chuang created HADOOP-16452: Summary: Increase ipc.maximum.data.length default from 64MB to 128MB Key: HADOOP-16452 URL: https://issues.apache.org/jira/browse/HADOOP-16452 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 2.6.0 Reporter: Wei-Chiu Chuang Reason for bumping the default: Denser DataNodes are common. It is not uncommon to find a DataNode with > 7 million blocks these days. With such a high number of blocks, the block report message can exceed the 64mb limit (defined by ipc.maximum.data.length). The block reports are rejected, causing missing blocks in HDFS. We had to double this configuration value in order to work around the issue. We are seeing an increasing number of these cases. I think it's time to revisit some of these default values as the hardware evolves. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA
a word of caution. according to INFRA-18748, asf infra is going to be cutting out blind PR building. So we'll need to be sure that precommit integration works e.g. testing PRs because there's a JIRA that a whitelisted contributor has associated and put in patch available status. On Mon, Jul 22, 2019 at 1:06 PM Wei-Chiu Chuang wrote: > > Historically, Hadoop developers create patches and attache them to JIRA, > andthen the Yetus bot runs precommit against the patch in the JIRA. > > The Github PR is more convenient for code review and less hassle for > committers to merge a commit. I am proposing for the community to prefer > Github PR over the traditional patch-in-jira. This doesn't mean we will > reject the traditional way, but we can move gradually to the new way. > Additionally, update the Hadoop "How to contribute" wiki, and advertise > that Github PR is the preferred method. > > Thoughts? -- busbey - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16380) S3A tombstones can confuse empty directory status
[ https://issues.apache.org/jira/browse/HADOOP-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16380. - Resolution: Fixed Fixed -thanks for help in testing this > S3A tombstones can confuse empty directory status > - > > Key: HADOOP-16380 > URL: https://issues.apache.org/jira/browse/HADOOP-16380 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.3.0 > > > If S3AFileSystem does an S3 LIST restricted to a single object to see if a > directory is empty, and the single entry found has a tombstone marker (either > from an inconsistent DDB Table or from an eventually consistent LIST) then it > will consider the directory empty, _even if there is 1+ entry which is not > deleted_ > We need to make sure the calculation of whether a directory is empty or not > is resilient to this, efficiently. > It surfaces as an issue two places > * delete(path) (where it may make things worse) > * rename(src, dest), where a check is made for dest != an empty directory. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16451) Update jackson-databind to 2.9.9.1
Wei-Chiu Chuang created HADOOP-16451: Summary: Update jackson-databind to 2.9.9.1 Key: HADOOP-16451 URL: https://issues.apache.org/jira/browse/HADOOP-16451 Project: Hadoop Common Issue Type: Bug Reporter: Wei-Chiu Chuang https://nvd.nist.gov/vuln/detail/CVE-2019-12814 CVE-2019-12814 flags 2.9.9 as vulnerable. A new version 2.9.9.1 is available. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16450) ITestS3ACommitterFactory failing, S3 client is not inconsistent
Steve Loughran created HADOOP-16450:
---
Summary: ITestS3ACommitterFactory failing, S3 client is not
inconsistent
Key: HADOOP-16450
URL: https://issues.apache.org/jira/browse/HADOOP-16450
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Transient failure of {{ITestS3ACommitterFactory}} during a sequential run; the
FS created wasn't inconsistent
That test suite doesn't override the superclass AbstractCommitITest's
{{useInconsistentClient}} method, so declares that it expects one. If we have
it return false, it won't care any more what kind of FS client it gets
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA
Thanks Wei-Chiu for starting this discussion. +1 (non-binding) for turning code review to GitHub PR and keep other comments/discussions on JIRA. 1. In my experience, JIRA is better at tracking and recording information. 2. It is confused that no guide (`How to contribute`) about submit PR to JIRA or GitHub, so there are a few patches and review comments active at both side. In my opinion it is necessary to unify them. Thanks, He Xiaoqiao On Tue, Jul 23, 2019 at 6:01 PM Gabor Bota wrote: > Although we will use github with PRs, I'd still prefer adding a +1 as a > jira comment stating which PR was the last and approved one among the many. > > On Tue, Jul 23, 2019 at 11:22 AM Steve Loughran > > wrote: > > > On Mon, Jul 22, 2019 at 7:29 PM Eric Badger > > wrote: > > > > > Where would JIRA fit into the PR workflow? Would we file JIRAs just to > > > track github PRs and have all of the discussion on the PR? > > > > > > > > Every code contribution needs its JIRA for: tracking, release notes, > cross > > referencing; every committed patch needs that JIRA reference. > > > > Reviews of specific patches go into the PRs > > > > I actually think discussion about overall direction of work is better in > > the JIRA, because a complex piece of work can have multiple PRs: > different > > attempts where when you need to rebase its best to create a new one so > the > > old discussion is still linked to specific lines of code, and when > > different people take a PR and contribute their own work. > > > > That split of comments across >1 PR is one of the costs of using github > for > > review. > > >
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/ [Jul 23, 2019 1:36:43 AM] (aajisaka) MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.fs.viewfs.TestViewFsURIs hadoop.fs.viewfs.TestFcMainOperationsLocalFs hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-compile-cc-root-jdk1.8.0_212.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-compile-javac-root-jdk1.8.0_212.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_212.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [184K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/patch-unit-hadoop-common-project_hadoop-nfs.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [28K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/391/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt [0]
Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA
Although we will use github with PRs, I'd still prefer adding a +1 as a jira comment stating which PR was the last and approved one among the many. On Tue, Jul 23, 2019 at 11:22 AM Steve Loughran wrote: > On Mon, Jul 22, 2019 at 7:29 PM Eric Badger > wrote: > > > Where would JIRA fit into the PR workflow? Would we file JIRAs just to > > track github PRs and have all of the discussion on the PR? > > > > > Every code contribution needs its JIRA for: tracking, release notes, cross > referencing; every committed patch needs that JIRA reference. > > Reviews of specific patches go into the PRs > > I actually think discussion about overall direction of work is better in > the JIRA, because a complex piece of work can have multiple PRs: different > attempts where when you need to rebase its best to create a new one so the > old discussion is still linked to specific lines of code, and when > different people take a PR and contribute their own work. > > That split of comments across >1 PR is one of the costs of using github for > review. >
Re: Aug Hadoop Community Meetup in China
Hi Junping Thanks. I would like to get a slot to talk about our new open source project: YuniKorn. Thanks Weiwei On Jul 23, 2019, 5:08 PM +0800, 俊平堵 , wrote: > Thanks for these positive feedbacks! The local community has voted the date > and location to be 8/10, Beijing. So please book your time ahead if you have > interest to join. > I have gathered a few topics, and some candidate places for hosting this > meetup. If you would like to propose more topics, please nominate it here or > ping me before this weekend (7/28, CST time). > Will update here when I have more to share. thx! > > > > > <> > > <> > > > > Thanks, > > Junping > > > 俊平堵 于2019年7月18日周四 下午3:28写道： > > > Hi, all! > > > I am glad to let you know that we are organizing Hadoop Contributors > > > Meetup in China on Aug. > > > > > > This could be the first time hadoop community meetup in China and many > > > attendees are expected to come from big data pioneers, such as: Cloudera, > > > Tencent, Alibaba, Xiaomi, Didi, JD, Meituan, Toutiao, Sina, etc. > > > > > > We're still working out the details, such as dates, contents and > > > locations. Here is a quick survey: https://www.surveymonkey.com/r/Y99RT3W > > > where you can vote your prefer dates and locations if you would like to > > > attend - the survey will end in July. 21. 12PM China Standard Time, and > > > result will go public in next day. > > > > > > Also, please feel free to reach out to me if you have a topic to propose > > > for the meetup. Will send out an update later with more details when I > > > get more to share. Thanks! > > > > > > Cheers, > > > > > > Junping
[jira] [Resolved] (HADOOP-16446) Rolling upgrade to Hadoop 3.2.0 breaks due to backward in-compatible change
[
https://issues.apache.org/jira/browse/HADOOP-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16446.
-
Resolution: Won't Fix
I'm going to have to close this as a wontfix. I Feel your pain -I really do, as
I've hit it [in my own code|https://github.com/steveloughran/cloudstore]. It's
just we needed to move on, finally.
sorry
> Rolling upgrade to Hadoop 3.2.0 breaks due to backward in-compatible change
> ---
>
> Key: HADOOP-16446
> URL: https://issues.apache.org/jira/browse/HADOOP-16446
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: xia0c
>Priority: Major
>
> Hi,
> When I try to update Hadoop-common to the last version 3.2.0, it breaks
> backward compatibility due to compile dependency change in commons.lang. This
> also breaks rolling upgrades since any client implementing this - like Apache
> Crunch.
> -The following code will fail to run with the error
> "java.lang.NoClassDefFoundError:
> org/apache/commons/lang/SerializationException":
>
> {code:java}
> public void Demo(){
> PCollection data = MemPipeline.typedCollectionOf(strings(), "a");
> }
> {code}
> Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA
On Mon, Jul 22, 2019 at 7:29 PM Eric Badger wrote: > Where would JIRA fit into the PR workflow? Would we file JIRAs just to > track github PRs and have all of the discussion on the PR? > > Every code contribution needs its JIRA for: tracking, release notes, cross referencing; every committed patch needs that JIRA reference. Reviews of specific patches go into the PRs I actually think discussion about overall direction of work is better in the JIRA, because a complex piece of work can have multiple PRs: different attempts where when you need to rebase its best to create a new one so the old discussion is still linked to specific lines of code, and when different people take a PR and contribute their own work. That split of comments across >1 PR is one of the costs of using github for review.
Re: Aug Hadoop Community Meetup in China
Thanks for these positive feedbacks! The local community has voted the date and location to be 8/10, Beijing. So please book your time ahead if you have interest to join. I have gathered a few topics, and some candidate places for hosting this meetup. If you would like to propose more topics, please nominate it here or ping me before this weekend (7/28, CST time). Will update here when I have more to share. thx! [image: Screen Shot 2019-07-23 at 10.15.54.png] [image: Screen Shot 2019-07-23 at 10.16.06.png] Thanks, Junping 俊平堵 于2019年7月18日周四 下午3:28写道： > Hi, all! > > I am glad to let you know that we are organizing > Hadoop Contributors Meetup in China on Aug. > > > This could be the first time hadoop community meetup in China and many > attendees are expected to come from big data pioneers, such as: Cloudera, > Tencent, Alibaba, Xiaomi, Didi, JD, Meituan, Toutiao, Sina, etc. > > > We're still working out the details, such as dates, contents and > locations. Here is a quick survey: https://www.surveymonkey.com/r/Y99RT3W > where you can vote your prefer dates and locations if you would like to > attend - the survey will end in July. 21. 12PM China Standard Time, and > result will go public in next day. > > > Also, please feel free to reach out to me if you have a topic to propose > for the meetup. Will send out an update later with more details when I get > more to share. Thanks! > > > Cheers, > > > Junping >
[jira] [Created] (HADOOP-16449) Allow an empty credential provider chain, separate chains for S3 and DDB
Siddharth Seth created HADOOP-16449: --- Summary: Allow an empty credential provider chain, separate chains for S3 and DDB Key: HADOOP-16449 URL: https://issues.apache.org/jira/browse/HADOOP-16449 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Siddharth Seth Assignee: Siddharth Seth Currently, credentials cannot be empty (falls back to using the default chain). Credentials for S3 and DDB are always the same. In some cases it can be useful to use a different credential chain for S3 and DDB, as well as allow for an empty credential chain. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16448) Connection to Hadoop homepage is not secure
Kaspar Tint created HADOOP-16448: Summary: Connection to Hadoop homepage is not secure Key: HADOOP-16448 URL: https://issues.apache.org/jira/browse/HADOOP-16448 Project: Hadoop Common Issue Type: Bug Components: website Affects Versions: 2.7.7, 3.2.0, 3.1.0 Reporter: Kaspar Tint Attachments: Screen Shot 2019-07-23 at 9.37.54 AM.png When visiting the Hadoop website with the latest Firefox browser (v 68.0.1) it appears that the website cannot be reached through secure means by default. The culprit seems to be the fact that the two header images presented on the page are loaded in via *HTTP* !Screen Shot 2019-07-23 at 9.37.54 AM.png!. These images are located in the respective locations: http://hadoop.apache.org/images/hadoop-logo.jpg http://www.apache.org/images/asf_logo_wide.png These images can be reached also from the following locations: https://hadoop.apache.org/images/hadoop-logo.jpg https://www.apache.org/images/asf_logo_wide.png As one can see, a fix could be made to use a more safe way of including in the two header pictures to the page. I feel like I am in danger when reading the Hadoop documentation from the official Hadoop webpage in a non secure way. Thus I felt the need to open this ticket and raise the issue in order to have a future where everyone can learn from Hadoop documentation in a safe and secure way. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
