[
https://issues.apache.org/jira/browse/MAPREDUCE-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428306#comment-15428306
]
Jason Lowe commented on MAPREDUCE-6761:
---------------------------------------
Thanks for the report and patch, [~pvary]! Apologies this was missed in
MAPREDUCE-6473.
One potential issue is if a user explicitly placed a service provider ahead of
the built-in service providers with the intent of overriding them. If that
provider doesn't load then we will log a message but proceed anyway. At that
point we should have failed, as would have even pre-MAPREDUCE-6473, but we end
up proceeding without using the user's expected protocol provider. The reason
we didn't see this kind of error before MAPREDUCE-6473 is because the error
occurs _after_ a "good" protocol provider, so we never tried to load it. After
MAPREDUCE-6473 all the protocol providers are loaded up front, so we found the
latent error.
So even after this proposed fix there's a potentially unexpected change in
behavior. To preserve the original behavior, initProviderList should stop
iterating the service providers when one of them throws. Then we can proceed
to the Cluster code that iterates the providers array we were able to load. If
one of them fits then we're good, otherwise we throw.
Other comments on the patch:
The core dependency on common creates a cycle because common already depends
upon core:
{noformat}
[ERROR] The projects in the reactor contain a cyclic reference: Edge between
'Vertex{label='org.apache.hadoop:hadoop-mapreduce-client-common:3.0.0-alpha2-SNAPSHOT'}'
and
'Vertex{label='org.apache.hadoop:hadoop-mapreduce-client-core:3.0.0-alpha2-SNAPSHOT'}'
introduces to cycle in the graph
org.apache.hadoop:hadoop-mapreduce-client-core:3.0.0-alpha2-SNAPSHOT -->
org.apache.hadoop:hadoop-mapreduce-client-common:3.0.0-alpha2-SNAPSHOT -->
org.apache.hadoop:hadoop-mapreduce-client-core:3.0.0-alpha2-SNAPSHOT -> [Help 1]
{noformat}
Not sure why the precommit build didn't catch this. Should be trivial to
create a stubbed protocol provider class in the test that always fits and use
that instead to remove the dependency.
Nit: It's unnecessary to add the client protocol provider as a field in the
Cluster object just for testing. If we use a custom protocol provider for test
as suggested above, that protocol provider can return a custom client protocol
class that we can test for via the existing Cluster#getClient method to know
whether it used the expected protocol provider. Not a must-fix.
> Regression when handling providers - invalid configuration
> ServiceConfiguration causes Cluster initialization failure
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6761
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 3.0.0-alpha2
> Reporter: Peter Vary
> Assignee: Peter Vary
> Labels: supportability
> Attachments: MAPREDUCE-6761.2.patch, MAPREDUCE-6761.patch
>
>
> When a rogue org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider
> defines a provider that is not on classpath, then the initialization is
> failed with the following exception:
> java.util.ServiceConfigurationError:
> org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider: Provider
> org.apache.hadoop.mapred.YarnClientProtocolProvider not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:239)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at org.apache.hadoop.mapreduce.Cluster.initProviderList(Cluster.java:84)
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:114)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:108)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:101)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:477)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
> This regression is caused by MAPREDUCE-6473
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
