[jira] [Commented] (HADOOP-15372) Race conditions and possible leaks in the Shell class
[
https://issues.apache.org/jira/browse/HADOOP-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450470#comment-16450470
]
Miklos Szegedi commented on HADOOP-15372:
-
Thank you for the patch [~ebadger].
{code:java}
shell.getProcess().destroy();{code}
I see that you use SIGTERM instead of SIGKILL. This may still leak child
processes.
{code:java}
destroyShellProcesses(getAllShells());
if(!exec.awaitTermination(10, TimeUnit.SECONDS)) {
destroyShellProcesses(getAllShells());
}{code}
This may still leak shells, if the creator thread is waiting itself on
something else and it creates the subshell, once the second {{getAllShells()}}
is called.
I have a general concern with the original patch that any other users of the
Shell class may need to change their code to use the same pattern. There are at
least 5 other users of Shell in the codebase.
Have you considered using process groups? Process groups may also solve the
case when the node manager signals the container localizer process with
SIGKILL. Currently, if this happens the shells will leak.
> Race conditions and possible leaks in the Shell class
> -
>
> Key: HADOOP-15372
> URL: https://issues.apache.org/jira/browse/HADOOP-15372
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Miklos Szegedi
>Assignee: Eric Badger
>Priority: Minor
> Attachments: HADOOP-15372.001.patch
>
>
> YARN-5641 introduced some cleanup code in the Shell class. It has a race
> condition. {{Shell.
> runCommand()}} can be called while/after {{Shell.getAllShells()}} returned
> all the shells to be cleaned up. This new thread can avoid the clean up, so
> that the process held by it can be leaked causing leaked localized files/etc.
> I see another issue as well. {{Shell.runCommand()}} has a finally block with
> a {{
> process.destroy();}} to clean up. However, the try catch block does not cover
> all instructions after the process is started, so for example we can exit the
> thread and leak the process, if {{
> timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15372) Race conditions and possible leaks in the Shell class
[
https://issues.apache.org/jira/browse/HADOOP-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15372:
Description:
YARN-5641 introduced some cleanup code in the Shell class. It has a race
condition. {{Shell.
runCommand()}} can be called while/after {{Shell.getAllShells()}} returned all
the shells to be cleaned up. This new thread can avoid the clean up, so that
the process held by it can be leaked causing leaked localized files/etc.
I see another issue as well. {{Shell.runCommand()}} has a finally block with a
{{
process.destroy();}} to clean up. However, the try catch block does not cover
all instructions after the process is started, so for example we can exit the
thread and leak the process, if {{
timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an exception.
was:
YARN-5641 introduced some cleanup code in the Shell class. It has a race
condition. {{Shell.
runCommand()}} can be called while/after {{Shell.getAllShells()}} returned all
the shells to be cleaned up. This new thread can avoid the clean up, so that
the process held by it can be leaked causing leaked localized files/etc.
I also see an unrelated issue as well. {{Shell.runCommand()}} has a finally
block with a {{
process.destroy();}} to clean up. However, the try catch block does not cover
all instructions after the process is started, so for example we can exit the
thread and leak the process, if {{
timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an exception.
> Race conditions and possible leaks in the Shell class
> -
>
> Key: HADOOP-15372
> URL: https://issues.apache.org/jira/browse/HADOOP-15372
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Miklos Szegedi
>Priority: Minor
>
> YARN-5641 introduced some cleanup code in the Shell class. It has a race
> condition. {{Shell.
> runCommand()}} can be called while/after {{Shell.getAllShells()}} returned
> all the shells to be cleaned up. This new thread can avoid the clean up, so
> that the process held by it can be leaked causing leaked localized files/etc.
> I see another issue as well. {{Shell.runCommand()}} has a finally block with
> a {{
> process.destroy();}} to clean up. However, the try catch block does not cover
> all instructions after the process is started, so for example we can exit the
> thread and leak the process, if {{
> timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15372) Race conditions and possible leaks in the Shell class
[
https://issues.apache.org/jira/browse/HADOOP-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429196#comment-16429196
]
Miklos Szegedi commented on HADOOP-15372:
-
[~ebadger], [~jlowe], what do you think? Is this a valid scenario?
> Race conditions and possible leaks in the Shell class
> -
>
> Key: HADOOP-15372
> URL: https://issues.apache.org/jira/browse/HADOOP-15372
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Miklos Szegedi
>Priority: Minor
>
> YARN-5641 introduced some cleanup code in the Shell class. It has a race
> condition. {{Shell.
> runCommand()}} can be called while/after {{Shell.getAllShells()}} returned
> all the shells to be cleaned up. This new thread can avoid the clean up, so
> that the process held by it can be leaked causing leaked localized files/etc.
> I see another issue as well. {{Shell.runCommand()}} has a finally block with
> a {{
> process.destroy();}} to clean up. However, the try catch block does not cover
> all instructions after the process is started, so for example we can exit the
> thread and leak the process, if {{
> timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15372) Race conditions and possible leaks in the Shell class
Miklos Szegedi created HADOOP-15372:
---
Summary: Race conditions and possible leaks in the Shell class
Key: HADOOP-15372
URL: https://issues.apache.org/jira/browse/HADOOP-15372
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.10.0, 3.2.0
Reporter: Miklos Szegedi
YARN-5641 introduced some cleanup code in the Shell class. It has a race
condition. {{Shell.
runCommand()}} can be called while/after {{Shell.getAllShells()}} returned all
the shells to be cleaned up. This new thread can avoid the clean up, so that
the process held by it can be leaked causing leaked localized files/etc.
I also see an unrelated issue as well. {{Shell.runCommand()}} has a finally
block with a {{
process.destroy();}} to clean up. However, the try catch block does not cover
all instructions after the process is started, so for example we can exit the
thread and leak the process, if {{
timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting
[ https://issues.apache.org/jira/browse/HADOOP-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated HADOOP-14855: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) > Hadoop scripts may errantly believe a daemon is still running, preventing it > from starting > -- > > Key: HADOOP-14855 > URL: https://issues.apache.org/jira/browse/HADOOP-14855 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha4 >Reporter: Aaron T. Myers >Assignee: Robert Kanter >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14855.001.patch, HADOOP-14855.002.patch > > > I encountered a case recently where the NN wouldn't start, with the error > message "namenode is running as process 16769. Stop it first." In fact the > NN was not running at all, but rather another long-running process was > running with this pid. > It looks to me like our scripts just check to see if _any_ process is running > with the pid that the NN (or any Hadoop daemon) most recently ran with. This > is clearly not a fool-proof way of checking to see if a particular type of > daemon is now running, as some other process could start running with the > same pid since the daemon in question was previously shut down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting
[ https://issues.apache.org/jira/browse/HADOOP-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426280#comment-16426280 ] Miklos Szegedi commented on HADOOP-14855: - Committed to trunk. Thank you for the patch [~rkanter] and for the reviews [~aw], [~atm] and [~ste...@apache.org]. > Hadoop scripts may errantly believe a daemon is still running, preventing it > from starting > -- > > Key: HADOOP-14855 > URL: https://issues.apache.org/jira/browse/HADOOP-14855 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha4 >Reporter: Aaron T. Myers >Assignee: Robert Kanter >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14855.001.patch, HADOOP-14855.002.patch > > > I encountered a case recently where the NN wouldn't start, with the error > message "namenode is running as process 16769. Stop it first." In fact the > NN was not running at all, but rather another long-running process was > running with this pid. > It looks to me like our scripts just check to see if _any_ process is running > with the pid that the NN (or any Hadoop daemon) most recently ran with. This > is clearly not a fool-proof way of checking to see if a particular type of > daemon is now running, as some other process could start running with the > same pid since the daemon in question was previously shut down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting
[ https://issues.apache.org/jira/browse/HADOOP-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426271#comment-16426271 ] Miklos Szegedi commented on HADOOP-14855: - +1 LGTM. I will commit this shortly. > Hadoop scripts may errantly believe a daemon is still running, preventing it > from starting > -- > > Key: HADOOP-14855 > URL: https://issues.apache.org/jira/browse/HADOOP-14855 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha4 >Reporter: Aaron T. Myers >Assignee: Robert Kanter >Priority: Major > Attachments: HADOOP-14855.001.patch, HADOOP-14855.002.patch > > > I encountered a case recently where the NN wouldn't start, with the error > message "namenode is running as process 16769. Stop it first." In fact the > NN was not running at all, but rather another long-running process was > running with this pid. > It looks to me like our scripts just check to see if _any_ process is running > with the pid that the NN (or any Hadoop daemon) most recently ran with. This > is clearly not a fool-proof way of checking to see if a particular type of > daemon is now running, as some other process could start running with the > same pid since the daemon in question was previously shut down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: (was: HADOOP.15331.001.patch)
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP-15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: HADOOP-15331.001.patch
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP-15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408601#comment-16408601
]
Miklos Szegedi commented on HADOOP-15331:
-
Fixing checkstyle.
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP.15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: HADOOP.15331.001.patch
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP.15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Status: Patch Available (was: Open)
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0, 2.7.5, 2.8.3, 2.9.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: HADOOP-15331.000.patch
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Description:
There is a race condition in the way Hadoop handles the Configuration class.
The scenario is the following. Let's assume that there are two threads sharing
the same Configuration class. One adds some resources to the configuration,
while the other one clones it. Resources are loaded lazily in a deferred call
to {{loadResources()}}. If the cloning happens after adding the resources but
before parsing them, some temporary resources like input stream pointers are
cloned. Eventually both copies will load the input stream resources pointing to
the same input streams. One parses the input stream XML and closes it updating
it's own copy of the resource. The other one has another pointer to the same
input stream. When it tries to load it, it will crash with a stream closed
exception.
Here is an example unit test:
{code:java}
@Test
public void testResourceRace() {
InputStream is =
new BufferedInputStream(new ByteArrayInputStream(
"".getBytes()));
Configuration conf = new Configuration();
// Thread 1
conf.addResource(is);
// Thread 2
Configuration confClone = new Configuration(conf);
// Thread 2
confClone.get("firstParse");
// Thread 1
conf.get("secondParse");
}{code}
Example real world stack traces:
{code:java}
2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
parsing conf java.io.BufferedInputStream@7741d346
com.ctc.wstx.exc.WstxIOException: Stream closed
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
at
org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
at
org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
{code}
Another example:
{code:java}
2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
parsing conf java.io.BufferedInputStream@7741d346
com.ctc.wstx.exc.WstxIOException: Stream closed
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebApp.buildRedirectPath(RMWebApp.java:103)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebApp.getRedirectPath(RMWebApp.java:91)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:125)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407305#comment-16407305
]
Miklos Szegedi commented on HADOOP-15331:
-
Note: HADOOP-11274 had a more eager approach in the same code that had to be
reverted.
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() throws Exception {
> InputStream is = new InputStream() {
> InputStream is =
> new ByteArrayInputStream(
> "".getBytes());
> @Override
> public int read() throws IOException {
> return is.read();
> }
> };
> Configuration conf = new Configuration();
> conf.addResource(new BufferedInputStream(is));
> Configuration confClone = new Configuration(conf);
> confClone.get("firstParse");
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Summary: Race condition causing org.apache.hadoop.conf.Configuration: error
parsing conf java.io.BufferedInputStream (was: Race conditions caused by
org.apache.hadoop.conf.Configuration: error parsing conf
java.io.BufferedInputStream)
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() throws Exception {
> InputStream is = new InputStream() {
> InputStream is =
> new ByteArrayInputStream(
> "".getBytes());
> @Override
> public int read() throws IOException {
> return is.read();
> }
> };
> Configuration conf = new Configuration();
> conf.addResource(new BufferedInputStream(is));
> Configuration confClone = new Configuration(conf);
> confClone.get("firstParse");
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
>
[jira] [Created] (HADOOP-15331) Race conditions caused by org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
Miklos Szegedi created HADOOP-15331:
---
Summary: Race conditions caused by
org.apache.hadoop.conf.Configuration: error parsing conf
java.io.BufferedInputStream
Key: HADOOP-15331
URL: https://issues.apache.org/jira/browse/HADOOP-15331
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 3.0.0, 2.7.5, 2.8.3, 2.9.0, 3.1.0, 2.10.0
Reporter: Miklos Szegedi
Assignee: Miklos Szegedi
There is a race condition in the way Hadoop handles the Configuration class.
The scenario is the following. Let's assume that there are two threads sharing
the same Configuration class. One adds some resources to the configuration,
while the other one clones it. Resources are loaded lazily in a deferred call
to {{loadResources()}}. If the cloning happens after adding the resources but
before parsing them, some temporary resources like input stream pointers are
cloned. Eventually both copies will load the input stream resources pointing to
the same input streams. One parses the input stream XML and closes it updating
it's own copy of the resource. The other one has another pointer to the same
input stream. When it tries to load it, it will crash with a stream closed
exception.
Here is an example unit test:
{code:java}
@Test
public void testResourceRace() throws Exception {
InputStream is = new InputStream() {
InputStream is =
new ByteArrayInputStream(
"".getBytes());
@Override
public int read() throws IOException {
return is.read();
}
};
Configuration conf = new Configuration();
conf.addResource(new BufferedInputStream(is));
Configuration confClone = new Configuration(conf);
confClone.get("firstParse");
conf.get("secondParse");
}{code}
Example real world stack traces:
{code:java}
2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
parsing conf java.io.BufferedInputStream@7741d346
com.ctc.wstx.exc.WstxIOException: Stream closed
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
at
org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
at
org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
{code}
Another example:
{code:java}
2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
parsing conf java.io.BufferedInputStream@7741d346
com.ctc.wstx.exc.WstxIOException: Stream closed
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebApp.buildRedirectPath(RMWebApp.java:103)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebApp.getRedirectPath(RMWebApp.java:91)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:125)
at
[jira] [Commented] (HADOOP-15062) TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
[
https://issues.apache.org/jira/browse/HADOOP-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405521#comment-16405521
]
Miklos Szegedi commented on HADOOP-15062:
-
I reran the Jenkins job. [~yufeigu], there are no issues this time.
> TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
> ---
>
> Key: HADOOP-15062
> URL: https://issues.apache.org/jira/browse/HADOOP-15062
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15062.000.patch
>
>
> {code}
> [ERROR]
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Time
> elapsed: 0.478 s <<< FAILURE!
> java.lang.AssertionError: Unable to instantiate codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, is the required version of
> OpenSSL installed?
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec.init(TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java:43)
> {code}
> This happened due to the following openssl change:
> https://github.com/openssl/openssl/commit/ff4b7fafb315df5f8374e9b50c302460e068f188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque
[
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323437#comment-16323437
]
Miklos Szegedi commented on HADOOP-14597:
-
All right, thank you for the reply [~raviprak]!
> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been
> made opaque
>
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch,
> HADOOP-14597.02.patch, HADOOP-14597.03.patch, HADOOP-14597.04.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails
> with this error
> {code}[WARNING]
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
> In function ‘check_update_max_output_len’:
> [WARNING]
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
> error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING] ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2
> version is that many types have been made opaque, i.e. applications are no
> longer allowed to look inside the internals of the structures
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque
[
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321696#comment-16321696
]
Miklos Szegedi commented on HADOOP-14597:
-
I verified and the patch still compiles on branch-2 and an old version like
centos 6.4
> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been
> made opaque
>
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch,
> HADOOP-14597.02.patch, HADOOP-14597.03.patch, HADOOP-14597.04.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails
> with this error
> {code}[WARNING]
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
> In function ‘check_update_max_output_len’:
> [WARNING]
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
> error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING] ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2
> version is that many types have been made opaque, i.e. applications are no
> longer allowed to look inside the internals of the structures
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15060) TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky
[
https://issues.apache.org/jira/browse/HADOOP-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15060:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.1
2.10.0
3.1.0
Target Version/s: 3.1.0, 2.10.0, 3.0.1
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3.0 and branch-2. Thank you for the review
[~yufeigu]!
> TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky
> ---
>
> Key: HADOOP-15060
> URL: https://issues.apache.org/jira/browse/HADOOP-15060
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Fix For: 3.1.0, 2.10.0, 3.0.1
>
> Attachments: YARN-7553.000.patch
>
>
> {code}
> [ERROR]
> testFiniteGroupResolutionTime(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
> Time elapsed: 61.975 s <<< FAILURE!
> java.lang.AssertionError:
> Expected the logs to carry a message about command timeout but was:
> 2017-11-22 00:10:57,523 WARN security.ShellBasedUnixGroupsMapping
> (ShellBasedUnixGroupsMapping.java:getUnixGroups(181)) - unable to return
> groups for user foobarnonexistinguser
> PartialGroupNameException The user name 'foobarnonexistinguser' is not found.
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:275)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:178)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> at
> org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime(TestShellBasedUnixGroupsMapping.java:278)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque
[
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310046#comment-16310046
]
Miklos Szegedi commented on HADOOP-14597:
-
[~raviprak], do you have any concerns backporting this to branch-2?
> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been
> made opaque
>
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch,
> HADOOP-14597.02.patch, HADOOP-14597.03.patch, HADOOP-14597.04.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails
> with this error
> {code}[WARNING]
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
> In function ‘check_update_max_output_len’:
> [WARNING]
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
> error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING] ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2
> version is that many types have been made opaque, i.e. applications are no
> longer allowed to look inside the internals of the structures
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15122) Lock down version of doxia-module-markdown plugin
[
https://issues.apache.org/jira/browse/HADOOP-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15122:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.1
3.1.0
Status: Resolved (was: Patch Available)
Thank you for the contribution [~elek]. Committed to trunk and branch-3.0.
> Lock down version of doxia-module-markdown plugin
> -
>
> Key: HADOOP-15122
> URL: https://issues.apache.org/jira/browse/HADOOP-15122
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HADOOP-15122-trunk.002.patch, HADOOP-15122.001.patch
>
>
> Since HADOOP-14364 we have a SNAPSHOT dependency in the main pom.xml:
> {code}
> +
> + org.apache.maven.doxia
> + doxia-module-markdown
> + 1.8-SNAPSHOT
> +
> {code}
> Most probably because some feature was missing from last released doxia
> markdown module.
> -I propose to lock down the version and use a fixed instance from the
> snapshot version. -
> UPDATE: in the meantime doxia 1.8 has been released. Snapshot artifacts could
> not been dowloaded any more:
> 404:
> https://repository.apache.org/snapshots/org/apache/maven/doxia/doxia-module-markdown/1.8-SNAPSHOT/doxia-module-markdown-1.8-SNAPSHOT.jar
> IMHO Hadoop can be built without changing it to 1.8 fix version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15122) Lock down version of doxia-module-markdown plugin
[
https://issues.apache.org/jira/browse/HADOOP-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15122:
Target Version/s: 3.1.0, 3.0.1 (was: 3.1.0)
> Lock down version of doxia-module-markdown plugin
> -
>
> Key: HADOOP-15122
> URL: https://issues.apache.org/jira/browse/HADOOP-15122
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Attachments: HADOOP-15122-trunk.002.patch, HADOOP-15122.001.patch
>
>
> Since HADOOP-14364 we have a SNAPSHOT dependency in the main pom.xml:
> {code}
> +
> + org.apache.maven.doxia
> + doxia-module-markdown
> + 1.8-SNAPSHOT
> +
> {code}
> Most probably because some feature was missing from last released doxia
> markdown module.
> -I propose to lock down the version and use a fixed instance from the
> snapshot version. -
> UPDATE: in the meantime doxia 1.8 has been released. Snapshot artifacts could
> not been dowloaded any more:
> 404:
> https://repository.apache.org/snapshots/org/apache/maven/doxia/doxia-module-markdown/1.8-SNAPSHOT/doxia-module-markdown-1.8-SNAPSHOT.jar
> IMHO Hadoop can be built without changing it to 1.8 fix version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15122) Lock down version of doxia-module-markdown plugin
[
https://issues.apache.org/jira/browse/HADOOP-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306634#comment-16306634
]
Miklos Szegedi edited comment on HADOOP-15122 at 12/30/17 1:49 AM:
---
+1 LGTM. I will commit this shortly. The unit test issues are unrelated. I have
seen them on other Jenkins runs.
was (Author: miklos.szeg...@cloudera.com):
+1 LGTM. I will commit this shortly
> Lock down version of doxia-module-markdown plugin
> -
>
> Key: HADOOP-15122
> URL: https://issues.apache.org/jira/browse/HADOOP-15122
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Attachments: HADOOP-15122-trunk.002.patch, HADOOP-15122.001.patch
>
>
> Since HADOOP-14364 we have a SNAPSHOT dependency in the main pom.xml:
> {code}
> +
> + org.apache.maven.doxia
> + doxia-module-markdown
> + 1.8-SNAPSHOT
> +
> {code}
> Most probably because some feature was missing from last released doxia
> markdown module.
> -I propose to lock down the version and use a fixed instance from the
> snapshot version. -
> UPDATE: in the meantime doxia 1.8 has been released. Snapshot artifacts could
> not been dowloaded any more:
> 404:
> https://repository.apache.org/snapshots/org/apache/maven/doxia/doxia-module-markdown/1.8-SNAPSHOT/doxia-module-markdown-1.8-SNAPSHOT.jar
> IMHO Hadoop can be built without changing it to 1.8 fix version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15122) Lock down version of doxia-module-markdown plugin
[
https://issues.apache.org/jira/browse/HADOOP-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306634#comment-16306634
]
Miklos Szegedi commented on HADOOP-15122:
-
+1 LGTM. I will commit this shortly
> Lock down version of doxia-module-markdown plugin
> -
>
> Key: HADOOP-15122
> URL: https://issues.apache.org/jira/browse/HADOOP-15122
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Attachments: HADOOP-15122-trunk.002.patch, HADOOP-15122.001.patch
>
>
> Since HADOOP-14364 we have a SNAPSHOT dependency in the main pom.xml:
> {code}
> +
> + org.apache.maven.doxia
> + doxia-module-markdown
> + 1.8-SNAPSHOT
> +
> {code}
> Most probably because some feature was missing from last released doxia
> markdown module.
> -I propose to lock down the version and use a fixed instance from the
> snapshot version. -
> UPDATE: in the meantime doxia 1.8 has been released. Snapshot artifacts could
> not been dowloaded any more:
> 404:
> https://repository.apache.org/snapshots/org/apache/maven/doxia/doxia-module-markdown/1.8-SNAPSHOT/doxia-module-markdown-1.8-SNAPSHOT.jar
> IMHO Hadoop can be built without changing it to 1.8 fix version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267894#comment-16267894
]
Miklos Szegedi commented on HADOOP-14898:
-
Thank you [~elek] for the patch. Could you provide a patch file named
HADOOP-14898.000.patch against the latest trunk, so that the Jenkins job (asf
license, etc.) can run on it?
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release process is much more harder: in case of a Dockerfile error (which
> could be test on dockerhub only after the
[jira] [Updated] (HADOOP-15062) TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
[
https://issues.apache.org/jira/browse/HADOOP-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15062:
Attachment: HADOOP-15062.000.patch
> TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
> ---
>
> Key: HADOOP-15062
> URL: https://issues.apache.org/jira/browse/HADOOP-15062
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: HADOOP-15062.000.patch
>
>
> {code}
> [ERROR]
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Time
> elapsed: 0.478 s <<< FAILURE!
> java.lang.AssertionError: Unable to instantiate codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, is the required version of
> OpenSSL installed?
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec.init(TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java:43)
> {code}
> This happened due to the following openssl change:
> https://github.com/openssl/openssl/commit/ff4b7fafb315df5f8374e9b50c302460e068f188
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15062) TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
[
https://issues.apache.org/jira/browse/HADOOP-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi moved YARN-7554 to HADOOP-15062:
---
Key: HADOOP-15062 (was: YARN-7554)
Project: Hadoop Common (was: Hadoop YARN)
> TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9
> ---
>
> Key: HADOOP-15062
> URL: https://issues.apache.org/jira/browse/HADOOP-15062
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>
> {code}
> [ERROR]
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Time
> elapsed: 0.478 s <<< FAILURE!
> java.lang.AssertionError: Unable to instantiate codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, is the required version of
> OpenSSL installed?
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec.init(TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java:43)
> {code}
> This happened due to the following openssl change:
> https://github.com/openssl/openssl/commit/ff4b7fafb315df5f8374e9b50c302460e068f188
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15060) TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky
[
https://issues.apache.org/jira/browse/HADOOP-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15060:
Attachment: YARN-7553.000.patch
> TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky
> ---
>
> Key: HADOOP-15060
> URL: https://issues.apache.org/jira/browse/HADOOP-15060
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-7553.000.patch
>
>
> {code}
> [ERROR]
> testFiniteGroupResolutionTime(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
> Time elapsed: 61.975 s <<< FAILURE!
> java.lang.AssertionError:
> Expected the logs to carry a message about command timeout but was:
> 2017-11-22 00:10:57,523 WARN security.ShellBasedUnixGroupsMapping
> (ShellBasedUnixGroupsMapping.java:getUnixGroups(181)) - unable to return
> groups for user foobarnonexistinguser
> PartialGroupNameException The user name 'foobarnonexistinguser' is not found.
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:275)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:178)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> at
> org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime(TestShellBasedUnixGroupsMapping.java:278)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15060) TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky
[
https://issues.apache.org/jira/browse/HADOOP-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15060:
Status: Patch Available (was: Open)
> TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky
> ---
>
> Key: HADOOP-15060
> URL: https://issues.apache.org/jira/browse/HADOOP-15060
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-7553.000.patch
>
>
> {code}
> [ERROR]
> testFiniteGroupResolutionTime(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
> Time elapsed: 61.975 s <<< FAILURE!
> java.lang.AssertionError:
> Expected the logs to carry a message about command timeout but was:
> 2017-11-22 00:10:57,523 WARN security.ShellBasedUnixGroupsMapping
> (ShellBasedUnixGroupsMapping.java:getUnixGroups(181)) - unable to return
> groups for user foobarnonexistinguser
> PartialGroupNameException The user name 'foobarnonexistinguser' is not found.
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:275)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:178)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> at
> org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime(TestShellBasedUnixGroupsMapping.java:278)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14974) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation fails in trunk
[
https://issues.apache.org/jira/browse/HADOOP-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215957#comment-16215957
]
Miklos Szegedi commented on HADOOP-14974:
-
It looks like this was brought in by
{code}
$ git bisect bad
14b3c2695b152285ed2919f2aa5a25e247c47871 is the first bad commit
commit 14b3c2695b152285ed2919f2aa5a25e247c47871
Author: John Zhuge
Date: Mon Oct 16 13:29:29 2017 -0700
HADOOP-14954. MetricsSystemImpl#init should increment refCount when already
initialized. Contributed by John Zhuge.
{code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
> fails in trunk
> ---
>
> Key: HADOOP-14974
> URL: https://issues.apache.org/jira/browse/HADOOP-14974
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: John Zhuge
>Priority: Blocker
>
> {code}
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> QueueMetrics,q0=root already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:239)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:141)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:90)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:267)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:639)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:331)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:391)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:756)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1313)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:161)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testExcessReservationThanNodeManagerCapacity(TestContainerAllocation.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Created] (HADOOP-14974) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation fails in trunk
Miklos Szegedi created HADOOP-14974:
---
Summary:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
fails in trunk
Key: HADOOP-14974
URL: https://issues.apache.org/jira/browse/HADOOP-14974
Project: Hadoop Common
Issue Type: Bug
Reporter: Miklos Szegedi
Assignee: John Zhuge
Priority: Blocker
{code}
org.apache.hadoop.metrics2.MetricsException: Metrics source
QueueMetrics,q0=root already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:239)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:141)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:131)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:90)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:267)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:639)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:331)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:391)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:756)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1152)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1313)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:161)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:140)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:136)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testExcessReservationThanNodeManagerCapacity(TestContainerAllocation.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14897) Loosen compatibility guidelines for native dependencies
[ https://issues.apache.org/jira/browse/HADOOP-14897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177340#comment-16177340 ] Miklos Szegedi commented on HADOOP-14897: - Thank you, [~templedf], I do not have any more comments. +1 (non-binding) > Loosen compatibility guidelines for native dependencies > --- > > Key: HADOOP-14897 > URL: https://issues.apache.org/jira/browse/HADOOP-14897 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, native >Reporter: Chris Douglas >Assignee: Daniel Templeton >Priority: Blocker > Attachments: HADOOP-14897.001.patch, HADOOP-14897.002.patch > > > Within a major version, the compatibility guidelines forbid raising the > minimum required version of any native dependency or tool required to build > native components. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14897) Loosen compatibility guidelines for native dependencies
[ https://issues.apache.org/jira/browse/HADOOP-14897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176875#comment-16176875 ] Miklos Szegedi commented on HADOOP-14897: - bq. To use this example, I would not require that Hadoop compile and run against a particular build of openjdk. That's too restrictive. Thank you [~chris.douglas] for the feedback. Here is the use case I am thinking about. Let's say I run my cluster with Hadoop 5.1 compiling with Java7. Auditing Java8 takes lots of effort and potential issues, so in order to install 5.2 with minor bug fixes, I think I should not be required to install Java8. However, to follow your logic, if there is no public support for Java7 anymore a minor release should update to Java 8. That is indeed something to consider. > Loosen compatibility guidelines for native dependencies > --- > > Key: HADOOP-14897 > URL: https://issues.apache.org/jira/browse/HADOOP-14897 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, native >Reporter: Chris Douglas >Assignee: Daniel Templeton >Priority: Blocker > > Within a major version, the compatibility guidelines forbid raising the > minimum required version of any native dependency or tool required to build > native components. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14897) Loosen compatibility guidelines for native dependencies
[ https://issues.apache.org/jira/browse/HADOOP-14897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176747#comment-16176747 ] Miklos Szegedi commented on HADOOP-14897: - [~chris.douglas]. Given we are talking about external dependencies, we cannot assume that they use the same versioning standard. However, talking about Hadoop versioning standard, I would assume that a minor release requires only minor releases in dependencies. Let's be more specific with an example. If we ship Hadoop 5.0 with support for C++ standards before C++14, this requirement would assume that 5.1 will not add a requirement of C++14 as a minimum. In case of Java I would rephrase this question, would you allow Hadoop 2.9 ship with a requirement of Java 8, if 2.8 depended on Java 7 before? > Loosen compatibility guidelines for native dependencies > --- > > Key: HADOOP-14897 > URL: https://issues.apache.org/jira/browse/HADOOP-14897 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, native >Reporter: Chris Douglas >Assignee: Daniel Templeton >Priority: Blocker > > Within a major version, the compatibility guidelines forbid raising the > minimum required version of any native dependency or tool required to build > native components. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168222#comment-16168222 ] Miklos Szegedi commented on HADOOP-13714: - Thank you, [~templedf]. I see a small typo: "container executer" in the latest version. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Blocker > Attachments: Compatibility.pdf, HADOOP-13714.001.patch, > HADOOP-13714.002.patch, HADOOP-13714.003.patch, HADOOP-13714.004.patch, > HADOOP-13714.005.patch, HADOOP-13714.006.patch, HADOOP-13714.WIP-001.patch, > InterfaceClassification.pdf > > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166828#comment-16166828 ] Miklos Szegedi commented on HADOOP-13714: - [~Daniel Templeton] What I suggest is to call out that minor versions should not require new versions of native dependencies like cmake, gcc, g++, zlib and ssl. Probably I would not call out them specifically since your document is general, it is up to you. I would mention native dependencies. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Blocker > Attachments: Compatibility.pdf, HADOOP-13714.001.patch, > HADOOP-13714.002.patch, HADOOP-13714.003.patch, HADOOP-13714.004.patch, > HADOOP-13714.005.patch, HADOOP-13714.WIP-001.patch, > InterfaceClassification.pdf > > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166828#comment-16166828 ] Miklos Szegedi edited comment on HADOOP-13714 at 9/14/17 7:04 PM: -- [~templedf] What I suggest is to call out that minor versions should not require new versions of native dependencies like cmake, gcc, g++, zlib and ssl. Probably I would not call out them specifically since your document is general, it is up to you. I would mention native dependencies. was (Author: miklos.szeg...@cloudera.com): [~Daniel Templeton] What I suggest is to call out that minor versions should not require new versions of native dependencies like cmake, gcc, g++, zlib and ssl. Probably I would not call out them specifically since your document is general, it is up to you. I would mention native dependencies. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Blocker > Attachments: Compatibility.pdf, HADOOP-13714.001.patch, > HADOOP-13714.002.patch, HADOOP-13714.003.patch, HADOOP-13714.004.patch, > HADOOP-13714.005.patch, HADOOP-13714.WIP-001.patch, > InterfaceClassification.pdf > > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165113#comment-16165113 ] Miklos Szegedi commented on HADOOP-13714: - Thank you, [~steve_l] for the explanations. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Blocker > Attachments: Compatibility.pdf, HADOOP-13714.001.patch, > HADOOP-13714.002.patch, HADOOP-13714.003.patch, HADOOP-13714.004.patch, > HADOOP-13714.WIP-001.patch, InterfaceClassification.pdf > > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164954#comment-16164954 ] Miklos Szegedi commented on HADOOP-13714: - [~templedf], thank you for working on this. I read the document and I have a few comments and questions. It would be helpful to have a shorter version or summary at the beginning for general audience. It could make sense to add a glossary. It might not be obvious to everyone for example what is downstream. bq. A package, class, or member variable or method that is not annotated SHALL be interpreted as implicitly Private and Unstable. bq. In cases where no classifications are present, the protocols SHOULD be assumed to be Private and Stable. Please help me understand the difference between protocols and code, why that they are treated so much differently. I would assume a code that is not marked as unstable should be stable. bq. Each API has an API-specific version number. Any incompatible changes MUST increment the API version number. I would also elaborate, whether the old API versions should be supported or not. In what circumstances can a REST API version be deprecated or removed. bq. All audit log output SHALL be considered Public and Stable. Any change to the data format SHALL be considered an incompatible change. Since this is an audit log, I am assuming a security requirement may override this requirement. Let’s say a certain event was not logged before but it should be. bq. The environment variables consumed by Hadoop and the environment variables made accessible to applications through YARN SHALL be considered Public and Evolving. The developer community SHOULD limit changes to major releases. Just a note. Hadoop configuration is public and stable. Do environment variables passed to the AM need to be evolving? bq. The JVM requirements SHALL NOT change across minor releases within the same major release unless the JVM version in question becomes unsupported. The JVM version requirement MAY be different for different operating systems or even operating system releases. Regarding this let’s call out C compiler and and runtime requirements. It might be frustrating not to be able to compile a new minor version of Hadoop with a security fix on an existing tested configuration. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Blocker > Attachments: Compatibility.pdf, HADOOP-13714.001.patch, > HADOOP-13714.002.patch, HADOOP-13714.003.patch, HADOOP-13714.004.patch, > HADOOP-13714.WIP-001.patch, InterfaceClassification.pdf > > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13963) /bin/bash is hard coded in some of the scripts
[ https://issues.apache.org/jira/browse/HADOOP-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109433#comment-16109433 ] Miklos Szegedi commented on HADOOP-13963: - LGTM (non-binding). Thank you for the patch [~ajayydv]. [~aw] do you have time to take a look? > /bin/bash is hard coded in some of the scripts > -- > > Key: HADOOP-13963 > URL: https://issues.apache.org/jira/browse/HADOOP-13963 > Project: Hadoop Common > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Ajay Yadav > Attachments: HADOOP-13963.01.patch > > > /bin/bash is hard coded in some of the scripts. We should consider using > #!/usr/bin/env bash at the beginning instead. > Candidates: > hadoop-functions_test_helper.bash > start-build-env.sh > findHangingTest.sh > verify-xml.sh > hadoop_env_checks.sh -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
[ https://issues.apache.org/jira/browse/HADOOP-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830666#comment-15830666 ] Miklos Szegedi commented on HADOOP-13977: - Thank you, [~asuresh]. That helped. > IntelliJ Compilation error in ITUseMiniCluster.java > --- > > Key: HADOOP-13977 > URL: https://issues.apache.org/jira/browse/HADOOP-13977 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Sean Busbey > Attachments: build.log > > > The repro steps: > mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA > to update indexes, etc. > ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java > Error:(34, 28) java: package org.apache.hadoop.fs does not exist > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
[ https://issues.apache.org/jira/browse/HADOOP-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated HADOOP-13977: Attachment: build.log [~busbey], I attached the logs requested. > IntelliJ Compilation error in ITUseMiniCluster.java > --- > > Key: HADOOP-13977 > URL: https://issues.apache.org/jira/browse/HADOOP-13977 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Sean Busbey > Attachments: build.log > > > The repro steps: > mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA > to update indexes, etc. > ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java > Error:(34, 28) java: package org.apache.hadoop.fs does not exist > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
[ https://issues.apache.org/jira/browse/HADOOP-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819533#comment-15819533 ] Miklos Szegedi commented on HADOOP-13977: - I checked the root cause. Building in IntelliJ worked before > IntelliJ Compilation error in ITUseMiniCluster.java > --- > > Key: HADOOP-13977 > URL: https://issues.apache.org/jira/browse/HADOOP-13977 > Project: Hadoop Common > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Sean Busbey > > The repro steps: > mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA > to update indexes, etc. > ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java > Error:(34, 28) java: package org.apache.hadoop.fs does not exist > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
[ https://issues.apache.org/jira/browse/HADOOP-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated HADOOP-13977: Comment: was deleted (was: I verified and this was introduced in HADOOP-11656.) > IntelliJ Compilation error in ITUseMiniCluster.java > --- > > Key: HADOOP-13977 > URL: https://issues.apache.org/jira/browse/HADOOP-13977 > Project: Hadoop Common > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Sean Busbey > > The repro steps: > mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA > to update indexes, etc. > ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java > Error:(34, 28) java: package org.apache.hadoop.fs does not exist > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
[ https://issues.apache.org/jira/browse/HADOOP-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819529#comment-15819529 ] Miklos Szegedi commented on HADOOP-13977: - I verified and this was introduced in HADOOP-11656. > IntelliJ Compilation error in ITUseMiniCluster.java > --- > > Key: HADOOP-13977 > URL: https://issues.apache.org/jira/browse/HADOOP-13977 > Project: Hadoop Common > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Sean Busbey > > The repro steps: > mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA > to update indexes, etc. > ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java > Error:(34, 28) java: package org.apache.hadoop.fs does not exist > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
Miklos Szegedi created HADOOP-13977: --- Summary: IntelliJ Compilation error in ITUseMiniCluster.java Key: HADOOP-13977 URL: https://issues.apache.org/jira/browse/HADOOP-13977 Project: Hadoop Common Issue Type: Bug Reporter: Miklos Szegedi Assignee: Sean Busbey The repro steps: mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA to update indexes, etc. ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java Error:(34, 28) java: package org.apache.hadoop.fs does not exist ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13963) /bin/bash is hard coded in some of the scripts
[ https://issues.apache.org/jira/browse/HADOOP-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812696#comment-15812696 ] Miklos Szegedi commented on HADOOP-13963: - This bug was opened based on the discussion with [~aw] in YARN-6060. > /bin/bash is hard coded in some of the scripts > -- > > Key: HADOOP-13963 > URL: https://issues.apache.org/jira/browse/HADOOP-13963 > Project: Hadoop Common > Issue Type: Bug >Reporter: Miklos Szegedi > > /bin/bash is hard coded in some of the scripts. We should consider using > #!/usr/bin/env bash at the beginning instead. > Candidates: > hadoop-functions_test_helper.bash > start-build-env.sh > findHangingTest.sh > verify-xml.sh > hadoop_env_checks.sh -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13963) /bin/bash is hard coded in some of the scripts
Miklos Szegedi created HADOOP-13963: --- Summary: /bin/bash is hard coded in some of the scripts Key: HADOOP-13963 URL: https://issues.apache.org/jira/browse/HADOOP-13963 Project: Hadoop Common Issue Type: Bug Reporter: Miklos Szegedi /bin/bash is hard coded in some of the scripts. We should consider using #!/usr/bin/env bash at the beginning instead. Candidates: hadoop-functions_test_helper.bash start-build-env.sh findHangingTest.sh verify-xml.sh hadoop_env_checks.sh -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
