[jira] [Commented] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363715#comment-15363715
]
Hadoop QA commented on HADOOP-13283:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 5s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.TestCrcCorruption |
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816350/HADOOP-13283.004.patch
|
| JIRA Issue | HADOOP-13283 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7ddced19906f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d792a90 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (HADOOP-12064) [JDK8] Update guice version to 4.0
[ https://issues.apache.org/jira/browse/HADOOP-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363701#comment-15363701 ] Panagiotis Garefalakis commented on HADOOP-12064: - Guice-4.0 is breaking resourceManager webapp interface ( check exception below ). After rolling back to guice3.0 the exception disappears. Problem accessing /cluster/scheduler. Reason: com.google.inject.Scopes.isCircularProxy(Ljava/lang/Object;)Z Caused by: java.lang.NoSuchMethodError: com.google.inject.Scopes.isCircularProxy(Ljava/lang/Object;)Z at com.google.inject.servlet.ServletScopes$1$1.get(ServletScopes.java:123) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:119) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:606) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:294) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:558) at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363621#comment-15363621 ] Allen Wittenauer commented on HADOOP-13335: --- bq. It's adding some YARN specific variables - which no existing scripts would have setup. The YARN\_\* vars were introduced when YARN was committed to Hadoop. So they are very, very old at this point in time. In any case, users do know about them and use them. bq. What happens when someone wants to talk to both yarn and hdfs? hdfs-config.sh is not going to get invoked anywhere. There are no HDFS\_\* vars that change how the client operates. Everything there is solely server-side. So there is zero impact that hdfs-config.sh isn't processed in this context. bq. yarn jar is just confusing - and doesn't seem to be providing a lot of utility. Totally agree, which is why I'm trying to eliminate all the extra vars. As I said above, we can't do that for a long time though. 2.x is missing the infrastructure. 3.x has the infrastructure and we can finally mark them as deprecated. Which, again, means the first opportunity to make this work in a sane way isn't until 4.x. :( bq. Hopefully we can have more clarity on what yarn jar means, and how it should be handled in trunk. I'm not sure there is more clarity required. If a user wants/needs YARN\_\*, then they need to use 'yarn jar'. If they don't want to use that, then they have the *option* to use 'hadoop jar' or 'yarn yar'. Hadoop has decided to give users a warning that YARN\_\* opts are not supported or utilized when 'hadoop jar' is executed as they likely intended to use 'yarn jar' instead. Adding a var here to turn this message off when it's already possible to turn this message off via other but similar means is excessive. So I guess I'm technically -1 on branch-2 too. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13283:
---
Attachment: HADOOP-13283.004.patch
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13283.000.patch, HADOOP-13283.001.patch,
> HADOOP-13283.002.patch, HADOOP-13283.003.patch, HADOOP-13283.004.patch
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363564#comment-15363564 ] Siddharth Seth commented on HADOOP-13335: - bq. If anything wants to talk to yarn in any way/shape/form and use hadoop's bootstrap code, they really, really, really should be using yarn jar. Otherwise there is a risk that some things may get mis-configured. We won't be able to clean this up until 4.x. (3.x finally deprecates a lot of this craziness from the project split back that happened in 0.22...) I'm still not clear about the utility of yarn jar. It's adding some YARN specific variables - which no existing scripts would have setup. Other than that - it appears to behave the same as hadoop jar. What happens when someone wants to talk to both yarn and hdfs? hdfs-config.sh is not going to get invoked anywhere. For trunk, it may make sense to separate the 'jar' sub-command instead of clubbing it along with service specific sub-commands (appplications, top, rmadmin, nodemanager, etc) under the 'yarn' command. yarn jar is just confusing - and doesn't seem to be providing a lot of utility. For this specific case - Hive could unset the YARN options in it's main script. Will have to check about the innovations of 'hadoop jar' that happen from within a running jvm. Lets get this patch into branch-2 at least. Hopefully we can have more clarity on what yarn jar means, and how it should be handled in trunk. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363547#comment-15363547
]
Hadoop QA commented on HADOOP-13283:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 37s{color} | {color:orange} root: The patch generated 1 new + 176 unchanged
- 0 fixed = 177 total (was 176) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
40s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHDFSXAttr |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816315/HADOOP-13283.003.patch
|
| JIRA Issue | HADOOP-13283 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f283f107532d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9560f25 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363520#comment-15363520 ] Allen Wittenauer commented on HADOOP-13335: --- I'm pretty much -1 on changing this in trunk. The warning is there for a reason. If 3rd parties want to explicitly go against that advice, then they just need to unset the given YARN env vars prior to launching hadoop jar. Unsetting those prior to launching hadoop jar has no impact since they aren't referenced at all in that code path. Therefore, no functionality is lost. This is also much simpler than adding Yet Another Configuration Variable and puts the onus on the caller to do whatever they want to do without putting more code maintenance on Hadoop. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers
[
https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363493#comment-15363493
]
Yufei Gu commented on HADOOP-13254:
---
[~rkanter], thanks for the review. Yes, good point. If a {{DiskValidator}}
creates background threads, it is more like a daemon, and we'd better to have
separate functions to start it other than start it while creating, and may add
status(running, stop) to check as well. And we can add these once we'd like it
to be this way.
> Create framework for configurable disk checkers
> ---
>
> Key: HADOOP-13254
> URL: https://issues.apache.org/jira/browse/HADOOP-13254
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch,
> HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch,
> HADOOP-13254.006.patch, HADOOP-13254.007.patch, HADOOP-13254.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers
[
https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363437#comment-15363437
]
Hadoop QA commented on HADOOP-13254:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 26 unchanged - 1 fixed = 26 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
44s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816321/HADOOP-13254.008.patch
|
| JIRA Issue | HADOOP-13254 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d13d944af132 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9560f25 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9928/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9928/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create framework for configurable disk checkers
> ---
>
> Key: HADOOP-13254
> URL: https://issues.apache.org/jira/browse/HADOOP-13254
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch,
> HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch,
>
[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers
[
https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363388#comment-15363388
]
Robert Kanter commented on HADOOP-13254:
I'm concerned about this code in {{DiskValidatorFactory}}:
{code}
diskValidator = ReflectionUtils.newInstance(clazz, null);
// check the return of putIfAbsent() to see if any other thread
have put
// the instance with the same key into INSTANCES
DiskValidator diskValidatorRet =
INSTANCES.putIfAbsent(clazz, diskValidator);
if (diskValidatorRet != null) {
diskValidator = diskValidatorRet;
}
{code}
If the implementation of {{DiskValidator}} that is being created does something
like starting some threads, and {{putIfAbsent}} returns a previous
instantiation of the class, then we might have a problem. For example, if you
have a {{DiskValidator}} that starts off a background thread to poll SMART
status, you might end up with multiple background threads doing this.
> Create framework for configurable disk checkers
> ---
>
> Key: HADOOP-13254
> URL: https://issues.apache.org/jira/browse/HADOOP-13254
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch,
> HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch,
> HADOOP-13254.006.patch, HADOOP-13254.007.patch, HADOOP-13254.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13254) Create framework for configurable disk checkers
[ https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated HADOOP-13254: -- Attachment: HADOOP-13254.008.patch [~rchiang], thanks for the review. I uploaded patch 008 for all comments. > Create framework for configurable disk checkers > --- > > Key: HADOOP-13254 > URL: https://issues.apache.org/jira/browse/HADOOP-13254 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, > HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, > HADOOP-13254.006.patch, HADOOP-13254.007.patch, HADOOP-13254.008.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363358#comment-15363358 ] Siddharth Seth commented on HADOOP-13335: - [~aw] - when you get a chance, does this look good to go in ? > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363310#comment-15363310
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
14s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816308/HADOOP-13335.04.patch
|
| JIRA Issue | HADOOP-13335 |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 9c2444d14b40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9560f25 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9926/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9926/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch,
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13283:
---
Attachment: HADOOP-13283.003.patch
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13283.000.patch, HADOOP-13283.001.patch,
> HADOOP-13283.002.patch, HADOOP-13283.003.patch
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13254) Create framework for configurable disk checkers
[ https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363111#comment-15363111 ] Ray Chiang edited comment on HADOOP-13254 at 7/5/16 9:19 PM: - Changed summary to match code changes. I'm assuming the configuration settings will go into YARN-5137. More minor nits: - I see that DiskValidatorFactory#getInstance(String) has @param and @throws, but DiskValidatorFactory#getInstance(Class) does not. I know it's a bit redundant, but I see similar classes have the Javadoc for both methods. - Make the InterfaceAudience settings more conservative for now (since DiskChecker is @InterfaceAudience.Private): -- Keep DiskValidator as @InterfaceAudience.Private (i.e. Hadoop only) for now. -- For DiskValidatorFactory, either remove @InterfaceStability.Evolving or change it to @InterfaceAudience.Unstable. was (Author: rchiang): Changed summary to match code changes. I'm assuming the configuration settings will go into YARN-5137. More minor nits: - I see that DiskValidatorFactory#getInstance(String) has @param and @throws, but DiskValidatorFactory#getInstance(Class) does not. I know it's a bit redundant, but I see similar classes have the Javadoc for both methods. - Make the InterfaceAudience settings more conservative for now (since DiskChecker is @InterfaceAudience.Private): -- Keep DiskValidator as @InterfaceAudience.Private (i.e. Hadoop only) for now. -- For DiskValidatorFactory, remove @InterfaceStability.Evolving. > Create framework for configurable disk checkers > --- > > Key: HADOOP-13254 > URL: https://issues.apache.org/jira/browse/HADOOP-13254 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, > HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, > HADOOP-13254.006.patch, HADOOP-13254.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Status: Patch Available (was: Open) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Attachment: HADOOP-13335.04.patch > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Status: Open (was: Patch Available) Posted the patch based on master instead of trunk. Will upload another one shortly > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12747) support wildcard in libjars argument
[ https://issues.apache.org/jira/browse/HADOOP-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363237#comment-15363237 ] Sangjin Lee commented on HADOOP-12747: -- The latest version (v.7) should be good to review. FWIW, I tested it with a real cluster for local paths and non-local paths. > support wildcard in libjars argument > > > Key: HADOOP-12747 > URL: https://issues.apache.org/jira/browse/HADOOP-12747 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: HADOOP-12747.01.patch, HADOOP-12747.02.patch, > HADOOP-12747.03.patch, HADOOP-12747.04.patch, HADOOP-12747.05.patch, > HADOOP-12747.06.patch, HADOOP-12747.07.patch > > > There is a problem when a user job adds too many dependency jars in their > command line. The HADOOP_CLASSPATH part can be addressed, including using > wildcards (\*). But the same cannot be done with the -libjars argument. Today > it takes only fully specified file paths. > We may want to consider supporting wildcards as a way to help users in this > situation. The idea is to handle it the same way the JVM does it: \* expands > to the list of jars in that directory. It does not traverse into any child > directory. > Also, it probably would be a good idea to do it only for libjars (i.e. don't > do it for -files and -archives). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363222#comment-15363222
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13335 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816299/HADOOP-13335.03.patch
|
| JIRA Issue | HADOOP-13335 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9925/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch,
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.03_branch-2.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Attachment: HADOOP-13335.03.patch Renamed the parameter to be prefixed with HADOOP_. Also fixed a typo in the parameter name. The patch is not changing the log line. I'm going to skip changing the hadoop_error usage in this patch, and leave that instead for the jira which changes this throughout the script. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Assignee: Siddharth Seth Status: Patch Available (was: Open) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Attachment: HADOOP-13335.03_branch-2.patch > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Status: Open (was: Patch Available) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363172#comment-15363172 ] Allen Wittenauer commented on HADOOP-13335: --- bq. The problem comes in when YARN_CLIENT_OPTS / YARN_OPTS are defined on a shell where hive is invoked Unset them form the environment prior to launch. Those env vars aren't being used outside of the yarn script. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Status: Patch Available (was: Open) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Attachment: HADOOP-13335.02.patch Patch for master. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Attachment: HADOOP-13335.02_branch-2.patch The same patch - renamed to branch-2. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Status: Open (was: Patch Available) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363164#comment-15363164 ] Allen Wittenauer commented on HADOOP-13335: --- As to the patch: * for trunk, hadoop_error should be used instead of redirection * all env vars should start with either HADOOP_, YARN_, HDFS_, or MAPRED_. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363161#comment-15363161 ] Siddharth Seth commented on HADOOP-13335: - bq. But hive doesn't call yarn, it uses hadoop. hadoop does not import yarn-config.sh, which is where yarn configuration parameters are expected to be located. YARN_USER_CLASSPATH is not handled or likely defined at all. So none of those settings will be defined. Hive does not use 'yarn' at the moment (and hopefully will not need to). The problem comes in when YARN_CLIENT_OPTS / YARN_OPTS are defined on a shell where hive is invoked (for beeline / the cli for example). That ends up displaying the 'use yarn jar' message which is confusing. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363150#comment-15363150 ] Allen Wittenauer commented on HADOOP-13335: --- bq. If I'm not mistaken yarn-config.sh imports hadoop-config.sh - so parameters like HADOOP_USER_CLASSPATH_FIRST etc will be respected. YARN_CLASSPATH_FIRST is handled after building the hadoop classpath. But hive doesn't call yarn, it uses hadoop. hadoop does not import yarn-config.sh, which is where yarn configuration parameters are expected to be located. YARN\_USER\_CLASSPATH is not handled or likely defined *at all*. So none of those settings will be defined. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11361) Fix a race condition in MetricsSourceAdapter.updateJmxCache
[
https://issues.apache.org/jira/browse/HADOOP-11361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363120#comment-15363120
]
Yongjun Zhang commented on HADOOP-11361:
Hi [~brahmareddy],
I took a quick look at your latest patch,
You removed
{code}
lastRecs = null; // in case regular interval update is not running
{code}
and added
{code}
189 // Not making LastRecs null due to Race between multiple readers
and
190 // updater
190 lastRecsCleared = true;
{code}
Given that the code is doing {{lastRecsCleared = true;}}, the comment "Not
making LastRecs null ..." is a bit contradictory.
It looks to me that it's actually not a race condition issue (otherwise please
would you please explain what race condition you are seeing), because it looks
that a single thread would run into the same problem as explained in my comment
linked above. So seems to me that the solution would be to fix
{{MetricsSourceAdapter#updateAttrCache}} to let it check whether lastRecs is
NULL, and not to access it if so.
Does it make sense to you?
Thanks.
> Fix a race condition in MetricsSourceAdapter.updateJmxCache
> ---
>
> Key: HADOOP-11361
> URL: https://issues.apache.org/jira/browse/HADOOP-11361
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.1, 2.5.1, 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-111361-003.patch, HADOOP-11361-002.patch,
> HADOOP-11361-004.patch, HADOOP-11361.patch, HDFS-7487.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateAttrCache(MetricsSourceAdapter.java:247)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:177)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getAttribute(MetricsSourceAdapter.java:102)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363122#comment-15363122
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13335 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816280/HADOOP-13335.01.patch
|
| JIRA Issue | HADOOP-13335 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9924/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
> Attachments: HADOOP-13335.01.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Attachment: HADOOP-13335.01.patch Thanks for the explanation [~aw]. If I'm not mistaken yarn-config.sh imports hadoop-config.sh - so parameters like HADOOP_USER_CLASSPATH_FIRST etc will be respected. YARN_CLASSPATH_FIRST is handled after building the hadoop classpath. I'm not sure how conflicting parameters between HADOOP_CLIENT_OPTS and YARN_CLIENT_OPTS can be handled though. Anyway - from a Hive perspective - having the WARNING show if the user uses the same shell as the one used to execute a YARN command is quite confusing. Also given this was introduced in 2.7.0 - I think it's useful to add an option to disable the warning. Attaching a simple patch which allows the WARNING to be suppressed. Could you please take a look. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13339) MetricsSourceAdapter#updateAttrCache may throw NPE due to NULL lastRecs
[
https://issues.apache.org/jira/browse/HADOOP-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363113#comment-15363113
]
Yongjun Zhang commented on HADOOP-13339:
Thanks much [~ozawa]! Let me take a look at HADOOP-11361 patch too.
> MetricsSourceAdapter#updateAttrCache may throw NPE due to NULL lastRecs
> ---
>
> Key: HADOOP-13339
> URL: https://issues.apache.org/jira/browse/HADOOP-13339
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> The for loop below may find lastRecs NULL
> {code}
> private int updateAttrCache() {
> LOG.debug("Updating attr cache...");
> int recNo = 0;
> int numMetrics = 0;
> for (MetricsRecordImpl record : lastRecs) {
> for (MetricsTag t : record.tags()) {
> setAttrCacheTag(t, recNo);
> ++numMetrics;
> }
> for (AbstractMetric m : record.metrics()) {
> setAttrCacheMetric(m, recNo);
> ++numMetrics;
> }
> ++recNo;
> }
> LOG.debug("Done. # tags & metrics="+ numMetrics);
> return numMetrics;
> }
> {code}
> and throws NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-13335: Status: Patch Available (was: Open) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers
[ https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363111#comment-15363111 ] Ray Chiang commented on HADOOP-13254: - Changed summary to match code changes. I'm assuming the configuration settings will go into YARN-5137. More minor nits: - I see that DiskValidatorFactory#getInstance(String) has @param and @throws, but DiskValidatorFactory#getInstance(Class) does not. I know it's a bit redundant, but I see similar classes have the Javadoc for both methods. - Make the InterfaceAudience settings more conservative for now (since DiskChecker is @InterfaceAudience.Private): -- Keep DiskValidator as @InterfaceAudience.Private (i.e. Hadoop only) for now. -- For DiskValidatorFactory, remove @InterfaceStability.Evolving. > Create framework for configurable disk checkers > --- > > Key: HADOOP-13254 > URL: https://issues.apache.org/jira/browse/HADOOP-13254 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, > HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, > HADOOP-13254.006.patch, HADOOP-13254.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13254) Create framework for configurable disk checkers
[ https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-13254: Summary: Create framework for configurable disk checkers (was: Make Diskchecker Pluggable) > Create framework for configurable disk checkers > --- > > Key: HADOOP-13254 > URL: https://issues.apache.org/jira/browse/HADOOP-13254 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, > HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, > HADOOP-13254.006.patch, HADOOP-13254.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362984#comment-15362984 ] Allen Wittenauer edited comment on HADOOP-13335 at 7/5/16 6:42 PM: --- hadoop jar does not include any of the following yarn content that is normally initialized during the yarn-config.sh bootstrap: * anything defined in yarn-env.sh * some YARN\_\*\_DIR * some YARN\_\*\_OPT and YARN\_\*\_OPTS * YARN\_\*\_CLASSPATH If anything wants to talk to yarn in any way/shape/form and use hadoop's bootstrap code, they really, really, really should be using yarn jar. Otherwise there is a risk that some things may get mis-configured. We won't be able to clean this up until 4.x. (3.x finally deprecates a lot of this craziness from the project split back that happened in 0.22...) was (Author: aw): hadoop jar does not include any of the following yarn content that is normally initialized during the yarn-config.sh bootstrap: * anything defined in yarn-env.sh * some YARN_*_DIR * some YARN_*_OPT and YARN_*_OPTS * YARN_*_CLASSPATH If anything wants to talk to yarn in any way/shape/form and use hadoop's bootstrap code, they really, really, really should be using yarn jar. Otherwise there is a risk that some things may get mis-configured. We won't be able to clean this up until 4.x. (3.x finally deprecates a lot of this craziness from the project split back that happened in 0.22...) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362984#comment-15362984 ] Allen Wittenauer commented on HADOOP-13335: --- hadoop jar does not include any of the following yarn content that is normally initialized during the yarn-config.sh bootstrap: * anything defined in yarn-env.sh * some YARN_*_DIR * some YARN_*_OPT and YARN_*_OPTS * YARN_*_CLASSPATH If anything wants to talk to yarn in any way/shape/form and use hadoop's bootstrap code, they really, really, really should be using yarn jar. Otherwise there is a risk that some things may get mis-configured. We won't be able to clean this up until 4.x. (3.x finally deprecates a lot of this craziness from the project split back that happened in 0.22...) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362971#comment-15362971 ] Hudson commented on HADOOP-13342: - SUCCESS: Integrated in Hadoop-trunk-Commit #10051 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10051/]) HADOOP-13342. ISAL download is breaking the Dockerfile (raviprak: rev 9560f252cf1f6ac0f029f92f85fa710c191c177d) * dev-support/docker/Dockerfile > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362949#comment-15362949 ] Siddharth Seth commented on HADOOP-13335: - [~aw] - what does yarn jar mean (as compared to Hadoop jar) - is it's main intent to provide the yarn libraries in the classpath ? Will upload a patch a little later to optionally disable the warning. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362917#comment-15362917 ] Allen Wittenauer commented on HADOOP-13342: --- Thank you! > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-13342: -- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 Status: Resolved (was: Patch Available) Committed to trunk > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362898#comment-15362898
]
Hadoop QA commented on HADOOP-13342:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:85209cc |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816237/HADOOP-13342.00.patch
|
| JIRA Issue | HADOOP-13342 |
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 6a92038e325a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8e672e3 |
| shellcheck | v0.4.4 |
| modules | C: U: |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9923/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ISAL download is breaking the Dockerfile
>
>
> Key: HADOOP-13342
> URL: https://issues.apache.org/jira/browse/HADOOP-13342
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Attachments: HADOOP-13342.00.patch
>
>
> http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb
> is returning a 404. We need to replace or remove this hack to prevent this
> from happening in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362896#comment-15362896 ] Ravi Prakash commented on HADOOP-13342: --- +1 LGTM. Will commit > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362881#comment-15362881 ] Hadoop QA commented on HADOOP-13342: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9923/console in case of problems. > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13342: -- Status: Patch Available (was: Open) > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reassigned HADOOP-13342: - Assignee: Allen Wittenauer > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13342: -- Attachment: HADOOP-13342.00.patch -00: * remove ISAL > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13342) ISAL download is breaking the Dockerfile
Allen Wittenauer created HADOOP-13342: - Summary: ISAL download is breaking the Dockerfile Key: HADOOP-13342 URL: https://issues.apache.org/jira/browse/HADOOP-13342 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Allen Wittenauer Priority: Blocker http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb is returning a 404. We need to replace or remove this hack to prevent this from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13339) MetricsSourceAdapter#updateAttrCache may throw NPE due to NULL lastRecs
[
https://issues.apache.org/jira/browse/HADOOP-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362149#comment-15362149
]
Tsuyoshi Ozawa commented on HADOOP-13339:
-
Thanks [~yzhangal] for taking this issue.
{quote}
So it doesn't seem a pure synchronization issue.
{quote}
I think so too. I will comment it on HADOOP-11361, so please wait a moment.
> MetricsSourceAdapter#updateAttrCache may throw NPE due to NULL lastRecs
> ---
>
> Key: HADOOP-13339
> URL: https://issues.apache.org/jira/browse/HADOOP-13339
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> The for loop below may find lastRecs NULL
> {code}
> private int updateAttrCache() {
> LOG.debug("Updating attr cache...");
> int recNo = 0;
> int numMetrics = 0;
> for (MetricsRecordImpl record : lastRecs) {
> for (MetricsTag t : record.tags()) {
> setAttrCacheTag(t, recNo);
> ++numMetrics;
> }
> for (AbstractMetric m : record.metrics()) {
> setAttrCacheMetric(m, recNo);
> ++numMetrics;
> }
> ++recNo;
> }
> LOG.debug("Done. # tags & metrics="+ numMetrics);
> return numMetrics;
> }
> {code}
> and throws NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362107#comment-15362107 ] Yuming Wang commented on HADOOP-9631: - [~lohit], Can you please help me upload the [latest patch|https://drive.google.com/open?id=0BxL8Kzbd2N--QW5VaU9wX1o4Z0k], I'm not authorized. > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13341) Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT
[ https://issues.apache.org/jira/browse/HADOOP-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13341: -- Hadoop Flags: Incompatible change > Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT > -- > > Key: HADOOP-13341 > URL: https://issues.apache.org/jira/browse/HADOOP-13341 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > > Big features like YARN-2928 demonstrate that even senior level Hadoop > developers forget that daemons need a custom _OPT env var. We can replace > all of the custom vars with generic handling just like we do for the username > check. > For example, today: > HADOOP_NAMENODE_OPT > would become > HADOOP_namenode_OPT > But if I wanted custom distcp options, there is no equivalent. But if the > command replacement mode was, then > HADOOP_distcp_OPT > would automatically work. > In the case of YARN-2928, HADOOP_timelinereaderserver_OPT would automatically > be checked without any extra work on behalf of the contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13341) Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT
[ https://issues.apache.org/jira/browse/HADOOP-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13341: -- Description: Big features like YARN-2928 demonstrate that even senior level Hadoop developers forget that daemons need a custom _OPT env var. We can replace all of the custom vars with generic handling just like we do for the username check. For example, today: HADOOP_NAMENODE_OPT would become HADOOP_namenode_OPT But if I wanted custom distcp options, there is no equivalent. But if the command replacement mode was, then HADOOP_distcp_OPT would automatically work. In the case of YARN-2928, HADOOP_timelinereaderserver_OPT would automatically be checked without any extra work on behalf of the contributors. was: Big features like YARN-2928 demonstrate that even senior level Hadoop developers forget that daemons need a custom _OPT env var. We can replace all of the custom vars with generic handling just like we do for the username check. For example, today: HADOOP_NAMENODE_OPT would become HADOOP_namenode_OPT But if I wanted custom distcp options, there is no equivalent. But if the command replacement mode was, then HADOOP_distcp_OPT would automatically work. > Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT > -- > > Key: HADOOP-13341 > URL: https://issues.apache.org/jira/browse/HADOOP-13341 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer > > Big features like YARN-2928 demonstrate that even senior level Hadoop > developers forget that daemons need a custom _OPT env var. We can replace > all of the custom vars with generic handling just like we do for the username > check. > For example, today: > HADOOP_NAMENODE_OPT > would become > HADOOP_namenode_OPT > But if I wanted custom distcp options, there is no equivalent. But if the > command replacement mode was, then > HADOOP_distcp_OPT > would automatically work. > In the case of YARN-2928, HADOOP_timelinereaderserver_OPT would automatically > be checked without any extra work on behalf of the contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13341) Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT
[ https://issues.apache.org/jira/browse/HADOOP-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reassigned HADOOP-13341: - Assignee: Allen Wittenauer > Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT > -- > > Key: HADOOP-13341 > URL: https://issues.apache.org/jira/browse/HADOOP-13341 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > > Big features like YARN-2928 demonstrate that even senior level Hadoop > developers forget that daemons need a custom _OPT env var. We can replace > all of the custom vars with generic handling just like we do for the username > check. > For example, today: > HADOOP_NAMENODE_OPT > would become > HADOOP_namenode_OPT > But if I wanted custom distcp options, there is no equivalent. But if the > command replacement mode was, then > HADOOP_distcp_OPT > would automatically work. > In the case of YARN-2928, HADOOP_timelinereaderserver_OPT would automatically > be checked without any extra work on behalf of the contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13341) Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT
Allen Wittenauer created HADOOP-13341: - Summary: Deprecate HADOOP_SERVERNAME_OPT; replace with HADOOP_(command)_OPT Key: HADOOP-13341 URL: https://issues.apache.org/jira/browse/HADOOP-13341 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer Big features like YARN-2928 demonstrate that even senior level Hadoop developers forget that daemons need a custom _OPT env var. We can replace all of the custom vars with generic handling just like we do for the username check. For example, today: HADOOP_NAMENODE_OPT would become HADOOP_namenode_OPT But if I wanted custom distcp options, there is no equivalent. But if the command replacement mode was, then HADOOP_distcp_OPT would automatically work. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
