[jira] [Updated] (HADOOP-15477) Make unjar in RunJar overrideable
[
https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Gustavsson updated HADOOP-15477:
--
Attachment: HADOOP-15477.004.patch
> Make unjar in RunJar overrideable
> -
>
> Key: HADOOP-15477
> URL: https://issues.apache.org/jira/browse/HADOOP-15477
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2
>Reporter: Johan Gustavsson
>Priority: Trivial
> Attachments: HADOOP-15477.001.patch, HADOOP-15477.002.patch,
> HADOOP-15477.003.patch, HADOOP-15477.004.patch
>
>
> Currently Hadoop's RunJar will unjar the jar provided and look for any jars
> inside and add them to the classpath. Since most deployments doesn't use jar
> in jar, but rather uberjars this could be rather time consuming at times and
> can cause issues related to over consumption of inodes, for something that is
> in many cases is not used.
> For that purpose there should be an env variable to disable this behavior.
>
> Edit: As requested by [~ajisakaa] in person here is a more detailed
> description of the issues we are trying to solve with this.
> A good chunk of our workloads are packaged in an uberjar, and are launched as
> a separate process using the {{hadoop jar}} cli. This is has generally been
> working out pretty well historically, with sub second launch times and good
> client isolation. Since bumping the host OS to a version patched with
> Meltdown/Specter patches we do see from time to time load becoming very high
> even with only a few client processes running and a single unjar process
> taking up to 10min.
> While another simple approach would be to abandon using the {{hadoop jar}}
> cli this would most likely take a lot more work than simply disabling unjar
> for the time being.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15477) Make unjar in RunJar overrideable
[
https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486764#comment-16486764
]
Johan Gustavsson commented on HADOOP-15477:
---
[~ste...@apache.org], point take :)
Added some minor tests but they required some further changes to the RunJar
class...
> Make unjar in RunJar overrideable
> -
>
> Key: HADOOP-15477
> URL: https://issues.apache.org/jira/browse/HADOOP-15477
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2
>Reporter: Johan Gustavsson
>Priority: Trivial
> Attachments: HADOOP-15477.001.patch, HADOOP-15477.002.patch,
> HADOOP-15477.003.patch
>
>
> Currently Hadoop's RunJar will unjar the jar provided and look for any jars
> inside and add them to the classpath. Since most deployments doesn't use jar
> in jar, but rather uberjars this could be rather time consuming at times and
> can cause issues related to over consumption of inodes, for something that is
> in many cases is not used.
> For that purpose there should be an env variable to disable this behavior.
>
> Edit: As requested by [~ajisakaa] in person here is a more detailed
> description of the issues we are trying to solve with this.
> A good chunk of our workloads are packaged in an uberjar, and are launched as
> a separate process using the {{hadoop jar}} cli. This is has generally been
> working out pretty well historically, with sub second launch times and good
> client isolation. Since bumping the host OS to a version patched with
> Meltdown/Specter patches we do see from time to time load becoming very high
> even with only a few client processes running and a single unjar process
> taking up to 10min.
> While another simple approach would be to abandon using the {{hadoop jar}}
> cli this would most likely take a lot more work than simply disabling unjar
> for the time being.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15477) Make unjar in RunJar overrideable
[
https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Gustavsson updated HADOOP-15477:
--
Attachment: HADOOP-15477.003.patch
> Make unjar in RunJar overrideable
> -
>
> Key: HADOOP-15477
> URL: https://issues.apache.org/jira/browse/HADOOP-15477
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2
>Reporter: Johan Gustavsson
>Priority: Trivial
> Attachments: HADOOP-15477.001.patch, HADOOP-15477.002.patch,
> HADOOP-15477.003.patch
>
>
> Currently Hadoop's RunJar will unjar the jar provided and look for any jars
> inside and add them to the classpath. Since most deployments doesn't use jar
> in jar, but rather uberjars this could be rather time consuming at times and
> can cause issues related to over consumption of inodes, for something that is
> in many cases is not used.
> For that purpose there should be an env variable to disable this behavior.
>
> Edit: As requested by [~ajisakaa] in person here is a more detailed
> description of the issues we are trying to solve with this.
> A good chunk of our workloads are packaged in an uberjar, and are launched as
> a separate process using the {{hadoop jar}} cli. This is has generally been
> working out pretty well historically, with sub second launch times and good
> client isolation. Since bumping the host OS to a version patched with
> Meltdown/Specter patches we do see from time to time load becoming very high
> even with only a few client processes running and a single unjar process
> taking up to 10min.
> While another simple approach would be to abandon using the {{hadoop jar}}
> cli this would most likely take a lot more work than simply disabling unjar
> for the time being.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15477) Make unjar in RunJar overrideable
[
https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Gustavsson updated HADOOP-15477:
--
Description:
Currently Hadoop's RunJar will unjar the jar provided and look for any jars
inside and add them to the classpath. Since most deployments doesn't use jar in
jar, but rather uberjars this could be rather time consuming at times and can
cause issues related to over consumption of inodes, for something that is in
many cases is not used.
For that purpose there should be an env variable to disable this behavior.
Edit: As requested by [~ajisakaa] in person here is a more detailed description
of the issues we are trying to solve with this.
A good chunk of our workloads are packaged in an uberjar, and are launched as a
separate process using the {{hadoop jar}} cli. This is has generally been
working out pretty well historically, with sub second launch times and good
client isolation. Since bumping the host OS to a version patched with
Meltdown/Specter patches we do see from time to time load becoming very high
even with only a few client processes running and a single unjar process taking
up to 10min.
While another simple approach would be to abandon using the {{hadoop jar}} cli
this would most likely take a lot more work than simply disabling unjar for the
time being.
was:
Currently Hadoop's RunJar will unjar the jar provided and look for any jars
inside and add them to the classpath. Since most deployments doesn't use jar in
jar, but rather uberjars this could be rather time consuming at times and can
cause issues related to over consumption of inodes, for something that is in
many cases is not used.
For that purpose there should be an env variable to disable this behavior.
> Make unjar in RunJar overrideable
> -
>
> Key: HADOOP-15477
> URL: https://issues.apache.org/jira/browse/HADOOP-15477
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2
>Reporter: Johan Gustavsson
>Priority: Trivial
> Attachments: HADOOP-15477.001.patch, HADOOP-15477.002.patch
>
>
> Currently Hadoop's RunJar will unjar the jar provided and look for any jars
> inside and add them to the classpath. Since most deployments doesn't use jar
> in jar, but rather uberjars this could be rather time consuming at times and
> can cause issues related to over consumption of inodes, for something that is
> in many cases is not used.
> For that purpose there should be an env variable to disable this behavior.
>
> Edit: As requested by [~ajisakaa] in person here is a more detailed
> description of the issues we are trying to solve with this.
> A good chunk of our workloads are packaged in an uberjar, and are launched as
> a separate process using the {{hadoop jar}} cli. This is has generally been
> working out pretty well historically, with sub second launch times and good
> client isolation. Since bumping the host OS to a version patched with
> Meltdown/Specter patches we do see from time to time load becoming very high
> even with only a few client processes running and a single unjar process
> taking up to 10min.
> While another simple approach would be to abandon using the {{hadoop jar}}
> cli this would most likely take a lot more work than simply disabling unjar
> for the time being.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15477) Make unjar in RunJar overrideable
[ https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480285#comment-16480285 ] Johan Gustavsson commented on HADOOP-15477: --- [~ajisakaa] could I ask you to take a look at this trivial patch or reassign it to whoever would be the right person to review it? Reason for no test is due to it's simplicity. > Make unjar in RunJar overrideable > - > > Key: HADOOP-15477 > URL: https://issues.apache.org/jira/browse/HADOOP-15477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2 >Reporter: Johan Gustavsson >Priority: Trivial > Attachments: HADOOP-15477.001.patch, HADOOP-15477.002.patch > > > Currently Hadoop's RunJar will unjar the jar provided and look for any jars > inside and add them to the classpath. Since most deployments doesn't use jar > in jar, but rather uberjars this could be rather time consuming at times and > can cause issues related to over consumption of inodes, for something that is > in many cases is not used. > For that purpose there should be an env variable to disable this behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15477) Make unjar in RunJar overrideable
[ https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated HADOOP-15477: -- Attachment: HADOOP-15477.002.patch > Make unjar in RunJar overrideable > - > > Key: HADOOP-15477 > URL: https://issues.apache.org/jira/browse/HADOOP-15477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2 >Reporter: Johan Gustavsson >Priority: Trivial > Attachments: HADOOP-15477.001.patch, HADOOP-15477.002.patch > > > Currently Hadoop's RunJar will unjar the jar provided and look for any jars > inside and add them to the classpath. Since most deployments doesn't use jar > in jar, but rather uberjars this could be rather time consuming at times and > can cause issues related to over consumption of inodes, for something that is > in many cases is not used. > For that purpose there should be an env variable to disable this behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15477) Make unjar in RunJar overrideable
[ https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated HADOOP-15477: -- Status: Patch Available (was: Open) > Make unjar in RunJar overrideable > - > > Key: HADOOP-15477 > URL: https://issues.apache.org/jira/browse/HADOOP-15477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.2, 2.7.6, 2.9.1, 2.8.3 >Reporter: Johan Gustavsson >Priority: Trivial > Attachments: HADOOP-15477.001.patch > > > Currently Hadoop's RunJar will unjar the jar provided and look for any jars > inside and add them to the classpath. Since most deployments doesn't use jar > in jar, but rather uberjars this could be rather time consuming at times and > can cause issues related to over consumption of inodes, for something that is > in many cases is not used. > For that purpose there should be an env variable to disable this behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15477) Make unjar in RunJar overrideable
[ https://issues.apache.org/jira/browse/HADOOP-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated HADOOP-15477: -- Attachment: HADOOP-15477.001.patch > Make unjar in RunJar overrideable > - > > Key: HADOOP-15477 > URL: https://issues.apache.org/jira/browse/HADOOP-15477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.3, 2.9.1, 2.7.6, 3.0.2 >Reporter: Johan Gustavsson >Priority: Trivial > Attachments: HADOOP-15477.001.patch > > > Currently Hadoop's RunJar will unjar the jar provided and look for any jars > inside and add them to the classpath. Since most deployments doesn't use jar > in jar, but rather uberjars this could be rather time consuming at times and > can cause issues related to over consumption of inodes, for something that is > in many cases is not used. > For that purpose there should be an env variable to disable this behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15477) Make unjar in RunJar overrideable
Johan Gustavsson created HADOOP-15477: - Summary: Make unjar in RunJar overrideable Key: HADOOP-15477 URL: https://issues.apache.org/jira/browse/HADOOP-15477 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.2, 2.7.6, 2.9.1, 2.8.3 Reporter: Johan Gustavsson Currently Hadoop's RunJar will unjar the jar provided and look for any jars inside and add them to the classpath. Since most deployments doesn't use jar in jar, but rather uberjars this could be rather time consuming at times and can cause issues related to over consumption of inodes, for something that is in many cases is not used. For that purpose there should be an env variable to disable this behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
