[jira] [Commented] (HADOOP-18142) Increase precommit job timeout from 24 hr to 30 hr
[ https://issues.apache.org/jira/browse/HADOOP-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497431#comment-17497431 ] Zoltan Haindrich commented on HADOOP-18142: --- [~vjasani] I've noticed this ticket by chance and would like to mention a few things I've done in Hive to avoid kinda like the same issues: * [use rateLimit|https://github.com/apache/hive/blob/af013246100be85675d18e6dcfcea7f202bc8d2c/Jenkinsfile#L21] to avoid building the same PR multiple time a day ; this naturally adds a 6 hour wait before the next would start * use a global lock to [limit the number|https://github.com/apache/hive/blob/af013246100be85675d18e6dcfcea7f202bc8d2c/Jenkinsfile#L150] of concurrently running * [disable concurrent builds|https://github.com/apache/hive/blob/af013246100be85675d18e6dcfcea7f202bc8d2c/Jenkinsfile#L23] as there is no point running the tests for someone who pushed new changes while it was still running => the contributor most likely will push more commits anyway which could launch even more builds...not starting a new build means it could pick up multiple trigger events while the one executing is still running * auto-kill the build in case the PR was updated while it waiting/running ; by calling [this method|https://github.com/apache/hive/blob/master/Jenkinsfile#L30-L45] at a few key points in the build > Increase precommit job timeout from 24 hr to 30 hr > -- > > Key: HADOOP-18142 > URL: https://issues.apache.org/jira/browse/HADOOP-18142 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > As per some recent precommit build results, full build QA is not getting > completed in 24 hr (recent example > [here|https://github.com/apache/hadoop/pull/4000] where more than 5 builds > timed out after 24 hr). We should increase it to 30 hr. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16582) LocalFileSystem's mkdirs() does not work as expected under viewfs.
[
https://issues.apache.org/jira/browse/HADOOP-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148345#comment-17148345
]
Zoltan Haindrich commented on HADOOP-16582:
---
[~ste...@apache.org], [~kihwal]: ping
> LocalFileSystem's mkdirs() does not work as expected under viewfs.
> --
>
> Key: HADOOP-16582
> URL: https://issues.apache.org/jira/browse/HADOOP-16582
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3, 3.1.3, 3.2.2
>
> Attachments: HADOOP-16582.1.patch, HADOOP-16582.patch
>
>
> When {{mkdirs(Path)}} is called against {{LocalFileSystem}}, the
> implementation in {{RawLocalFileSystem}} is called and the directory
> permission is determined by the umask. However, if it is under
> {{ViewFileSystem}}, the default implementation in {{FileSystem}} is called
> and this causes explicit {{chmod()}} to 0777.
> The {{mkdirs(Path)}} method needs to be overriden in
> - ViewFileSystem to avoid calling the default implementation
> - ChRootedFileSystem for proper resolution of viewfs mount table
> - FilterFileSystem to avoid calling the default implementation
> Only then the same method in the target ({{LocalFileSystem}} in this case)
> will be called. Hdfs does not suffer from the same flaw since it applies
> umask in all cases, regardless of what version of {{mkdirs()}} was called.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16582) LocalFileSystem's mkdirs() does not work as expected under viewfs.
[
https://issues.apache.org/jira/browse/HADOOP-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073843#comment-17073843
]
Zoltan Haindrich commented on HADOOP-16582:
---
[~ste...@apache.org], [~kihwal] this commit fundamentally changes how
{{mkdirs()}} work:
earlier:
* all plain {{mkdirs(somePath)}} were fast-tracked to {{FileSystem.mkdirs}}
which have rerouted them to {{mkdirs(somePath, somePerm)}} method with some
defaults (which were static)
* an implementation of {{FileSystem}} have only needed implement
"mkdirs(somePath, somePerm)" - because the other was not neccessarily called if
it was always in a {{FilterFileSystem}} or something like that
now:
* especially {{FilterFileSystem}} forwards the call of {{mkdirs(p)}} to the
actual fs implementation...which may skip overriden
{{mkdirs(somPath,somePerm)}} methods
* ...and could cause issues for existing FileSystem implementations
> LocalFileSystem's mkdirs() does not work as expected under viewfs.
> --
>
> Key: HADOOP-16582
> URL: https://issues.apache.org/jira/browse/HADOOP-16582
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3, 3.1.3, 3.2.2
>
> Attachments: HADOOP-16582.1.patch, HADOOP-16582.patch
>
>
> When {{mkdirs(Path)}} is called against {{LocalFileSystem}}, the
> implementation in {{RawLocalFileSystem}} is called and the directory
> permission is determined by the umask. However, if it is under
> {{ViewFileSystem}}, the default implementation in {{FileSystem}} is called
> and this causes explicit {{chmod()}} to 0777.
> The {{mkdirs(Path)}} method needs to be overriden in
> - ViewFileSystem to avoid calling the default implementation
> - ChRootedFileSystem for proper resolution of viewfs mount table
> - FilterFileSystem to avoid calling the default implementation
> Only then the same method in the target ({{LocalFileSystem}} in this case)
> will be called. Hdfs does not suffer from the same flaw since it applies
> umask in all cases, regardless of what version of {{mkdirs()}} was called.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16923) mkdir on s3a should not be sensitive to trailing '/'
Zoltan Haindrich created HADOOP-16923:
-
Summary: mkdir on s3a should not be sensitive to trailing '/'
Key: HADOOP-16923
URL: https://issues.apache.org/jira/browse/HADOOP-16923
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Zoltan Haindrich
I would have expected to create the directory for both calls:
{code}
[hive@hiveserver2-0 lib]$ hdfs dfs -mkdir
s3a://qe-s3-bucket-mst-xfpn-dwx-external/custom-jars2/
/usr/bin/hdfs: line 4: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such
file or directory
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
mkdir: get on s3a://qe-s3-bucket-mst-xfpn-dwx-external/custom-jars2/:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One or more
parameter values were invalid: An AttributeValue may not contain an empty
string (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ValidationException; Request ID:
1AE0KA2Q5ADI47R75N8BDJE973VV4KQNSO5AEMVJF66Q9ASUAAJG): One or more parameter
values were invalid: An AttributeValue may not contain an empty string
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;
Request ID: 1AE0KA2Q5ADI47R75N8BDJE973VV4KQNSO5AEMVJF66Q9ASUAAJG)
[hive@hiveserver2-0 lib]$ hdfs dfs -mkdir
s3a://qe-s3-bucket-mst-xfpn-dwx-external/custom-jars2
/usr/bin/hdfs: line 4: /usr/lib/bigtop-utils/bigtop-detect-javahome: No such
file or directory
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
[hive@hiveserver2-0 lib]$
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16908) Prune jackson 1 code again
[ https://issues.apache.org/jira/browse/HADOOP-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051864#comment-17051864 ] Zoltan Haindrich commented on HADOOP-16908: --- from the title; I would guess that you have already done this once :D for Hive we added a plugin to prevent similar things - there is a maven plugin which could check this at compile time: https://github.com/apache/hive/blob/deebfb630b809024dc29ee96391dd5104021802e/pom.xml#L1299 > Prune jackson 1 code again > -- > > Key: HADOOP-16908 > URL: https://issues.apache.org/jira/browse/HADOOP-16908 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > The jackson 1 code has silently creeped into the Hadoop codebase again. We > should prune them out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892514#comment-16892514 ] Zoltan Haindrich commented on HADOOP-16435: --- I don't think this is about a single session - these metrics are only used in the [IPC Server|https://github.com/apache/hadoop/blob/62deab17a33cef723d73f8d8b9e37e5bddbc1813/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L3075]. If you create an server and then shut it down - would you expect its metrics to be retained? (IMHO: if it silently retaines any information related to the actual instance that's a resource leak) > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891922#comment-16891922 ] Zoltan Haindrich edited comment on HADOOP-16435 at 7/24/19 2:54 PM: [~Jack-Lee] yes, it will be unregistered when the server is stopped - if I don't fully understand your concern; could you be more specific? was (Author: kgyrtkirk): [~Jack-Lee] yes, it will be unregistered when the server is stopped - if I don't fully understand your concern; could you elaborate? > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891922#comment-16891922 ] Zoltan Haindrich commented on HADOOP-16435: --- [~Jack-Lee] yes, it will be unregistered when the server is stopped - if I don't fully understand your concern; could you elaborate? > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16062) Add ability to disable Configuration reload registry
[
https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891822#comment-16891822
]
Zoltan Haindrich commented on HADOOP-16062:
---
[~aajisaka] could you please take a look?
note: it seems to me that the {{Configuration.reloadExistingConfigurations}}
functionality is only used to aid some interesting way to "deprecate" some
config keys
while keeping the deprecation disconnected from the main config keys section; I
think this could be probably done differently:
* by making sure that this "deprecation" call is either inside
{{org.apache.hadoop.fs.FileSystem}} implementations (or its imported)
* make sure that before constructing any "configuration" objects the
serviceloader is invoked - to ensure fully loading all FileSystem
implementation; which in turn will run all the static blocks
* what this is missing: probably for runtime loaded jars which contain
filesystem implementations this will not work
> Add ability to disable Configuration reload registry
>
>
> Key: HADOOP-16062
> URL: https://issues.apache.org/jira/browse/HADOOP-16062
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HADOOP-16062.01.patch, jxray_registry.png
>
>
> In Hive we see that after an extensive amount of usage there are a lot of
> Configuration objects not fully reclaimed because of Configuration's REGISTRY
> weak hashmap.
> https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321
> !jxray_registry.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HADOOP-16062: - Assignee: Zoltan Haindrich > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891774#comment-16891774 ] Zoltan Haindrich commented on HADOOP-16435: --- [~aajisaka] I've uploaded some more info; including a full heapdump somewhere around the heap gets filled up > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: related.jxray.png > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: rpcm.hprof.xz > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: defaultMetricsHoldsRpcMetrics.png > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: classes.png > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Status: Patch Available (was: Open) > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16435.01.patch > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: HADOOP-16435.01.patch > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16435.01.patch > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16435) RpcMetrics should not be retained forever
Zoltan Haindrich created HADOOP-16435: - Summary: RpcMetrics should not be retained forever Key: HADOOP-16435 URL: https://issues.apache.org/jira/browse/HADOOP-16435 Project: Hadoop Common Issue Type: Bug Components: rpc-server Reporter: Zoltan Haindrich * RpcMetrics objects are registered into [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] * although there is a shutdown() call (which is actually invoked) it doesn't unregisters itself from the [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] * RpcDetailedMetrics has the same issue background * hiveserver2 slowly eats up memory when running simple queries in new sessions (select 1) * every session opens a tezsession * tezsession has rpcmetrics * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884927#comment-16884927 ] Zoltan Haindrich commented on HADOOP-16062: --- [~elek], [~wangda], [~ste...@apache.org]: Could you please take a look? > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HADOOP-15205. --- Resolution: Fixed Fix Version/s: 3.2.0 3.1.2 both 3.1.2 and 3.2.0 has source attachments; I'm happy that this will be solved by the next hadoop upgrade. Thank you all! > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 3.0.1, 2.8.4, > 2.9.2, 2.8.5 >Reporter: Zoltan Haindrich >Priority: Major > Fix For: 3.1.2, 3.2.0 > > Attachments: chk.bash > > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749660#comment-16749660 ] Zoltan Haindrich commented on HADOOP-16062: --- [~elek] Could you please take a look? > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16062: -- Status: Patch Available (was: Open) > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748044#comment-16748044 ] Zoltan Haindrich commented on HADOOP-16062: --- this could be probably done a few different ways...I've attached a patch which does the above by setting the registry to null, in case it's disabled > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16062: -- Attachment: HADOOP-16062.01.patch > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16062) Add ability to disable Configuration reload registry
Zoltan Haindrich created HADOOP-16062: - Summary: Add ability to disable Configuration reload registry Key: HADOOP-16062 URL: https://issues.apache.org/jira/browse/HADOOP-16062 Project: Hadoop Common Issue Type: Improvement Reporter: Zoltan Haindrich In Hive we see that after an extensive amount of usage there are a lot of Configuration objects not fully reclaimed because of Configuration's REGISTRY weak hashmap. https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748020#comment-16748020 ] Zoltan Haindrich commented on HADOOP-16062: --- As these objects are belong to already closed sessions; they are just occupying heap space - and will force a "bigger" garbage collection eventually. I would like to propose to add a "toggle" to make it possible to disable this feature. > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16062: -- Description: In Hive we see that after an extensive amount of usage there are a lot of Configuration objects not fully reclaimed because of Configuration's REGISTRY weak hashmap. https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 !jxray_registry.png! was: In Hive we see that after an extensive amount of usage there are a lot of Configuration objects not fully reclaimed because of Configuration's REGISTRY weak hashmap. https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16062: -- Attachment: jxray_registry.png > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > Attachments: jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741395#comment-16741395 ] Zoltan Haindrich commented on HADOOP-15205: --- Thank you [~elek]! I think it will be a big step forward to have this fixed in future releases... > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 3.0.1, 2.8.4, > 2.9.2, 2.8.5 >Reporter: Zoltan Haindrich >Priority: Major > Attachments: chk.bash > > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602858#comment-16602858 ] Zoltan Haindrich commented on HADOOP-15205: --- [~ste...@apache.org] seems to me that 2.7.7 has source attachments! but unfortunately 3.1.1 doesn't seem to have http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.7/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.1/ I've written a small script ([^chk.bash]) which could run checks based on the projects in the maven build - I think that modules may come and go between releases...so having a project might not be good (but of course there might be some maven tricks I'm not aware of) It lists a few bogus results for even 2.7.7 (like hadoop-main and friends...which are logical modules) but for 3.1.1 source attachment is missing for example modules like: hadoop-yarn-services-api > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0, 3.1.0, 3.0.1 >Reporter: Zoltan Haindrich >Priority: Major > Attachments: chk.bash > > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-15205: -- Attachment: chk.bash > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0, 3.1.0, 3.0.1 >Reporter: Zoltan Haindrich >Priority: Major > Attachments: chk.bash > > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-15205: -- Environment: (was: 3.1.0 have also came out without sources as well... http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ I've added 3.1.1 as a target version...) Description: I wanted to use the source attachment; however it looks like since 2.7.5 that artifact is not present at maven central ; it looks like the last release which had source attachments / javadocs was 2.7.4 http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ this seems to be not limited to mapreduce; as the same change is present for yarn-common as well http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ and also hadoop-common http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ was: I wanted to use the source attachment; however it looks like since 2.7.5 that artifact is not present at maven central ; it looks like the last release which had source attachments / javadocs was 2.7.4 http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ this seems to be not limited to mapreduce; as the same change is present for yarn-common as well http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ and also hadoop-common http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ 3.1.0 have also came out without sources as well... http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ I've added 3.1.1 as a target version... > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0, 3.1.0, 3.0.1 >Reporter: Zoltan Haindrich >Priority: Major > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-15205: -- Affects Version/s: 3.1.0 3.0.1 Target Version/s: 3.1.1 Environment: 3.1.0 have also came out without sources as well... http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ I've added 3.1.1 as a target version... > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0, 3.1.0, 3.0.1 > Environment: 3.1.0 have also came out without sources as well... > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ > I've added 3.1.1 as a target version... >Reporter: Zoltan Haindrich >Priority: Major > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348394#comment-16348394 ] Zoltan Haindrich commented on HADOOP-15205: --- I think that having source attachments are extremely valuable during debug sessions; would it be possible to get them uploaded for the 3.0.0 release? > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0 >Reporter: Zoltan Haindrich >Priority: Major > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
Zoltan Haindrich created HADOOP-15205: - Summary: maven release: missing source attachments for hadoop-mapreduce-client-core Key: HADOOP-15205 URL: https://issues.apache.org/jira/browse/HADOOP-15205 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.7.5 Reporter: Zoltan Haindrich I wanted to use the source attachment; however it looks like since 2.7.5 that artifact is not present at maven central ; it looks like the last release which had source attachments / javadocs was 2.7.4 http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ this seems to be not limited to mapreduce; as the same change is present for yarn-common as well http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ and also hadoop-common http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[ https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260965#comment-16260965 ] Zoltan Haindrich commented on HADOOP-12760: --- Are there any chance this will be available in Hadoop-3.0 ? > sun.misc.Cleaner has moved to a new location in OpenJDK 9 > - > > Key: HADOOP-12760 > URL: https://issues.apache.org/jira/browse/HADOOP-12760 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chris Hegarty >Assignee: Akira Ajisaka > Attachments: HADOOP-12760.00.patch, HADOOP-12760.01.patch, > HADOOP-12760.02.patch, HADOOP-12760.03.patch > > > This is a heads-up: there are upcoming changes in JDK 9 that will require, at > least, a small update to org.apache.hadoop.crypto.CryptoStreamUtils & > org.apache.hadoop.io.nativeio.NativeIO. > OpenJDK issue no. 8148117: "Move sun.misc.Cleaner to jdk.internal.ref" [1], > will move the Cleaner class from sun.misc to jdk.internal.ref. There is > ongoing discussion about the possibility of providing a public supported API, > maybe in the JDK 9 timeframe, for releasing NIO direct buffer native memory, > see the core-libs-dev mail thread [2]. At the very least CryptoStreamUtils & > NativeIO [3] should be updated to have knowledge of the new location of the > JDK Cleaner. > [1] https://bugs.openjdk.java.net/browse/JDK-8148117 > [2] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/038243.html > [3] https://github.com/apache/hadoop/search?utf8=✓=sun.misc.Cleaner -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
