[jira] [Commented] (HADOOP-17214) Allow file system caching to be disabled for all file systems
[ https://issues.apache.org/jira/browse/HADOOP-17214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189846#comment-17189846 ] Haibo Chen commented on HADOOP-17214: - Linked https://issues.apache.org/jira/browse/HADOOP-13971 which contains the motivating problem for this new config > Allow file system caching to be disabled for all file systems > - > > Key: HADOOP-17214 > URL: https://issues.apache.org/jira/browse/HADOOP-17214 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > > Right now, FileSystem.get(URI uri, Configuration conf) allows caching of file > systems to be disabled per scheme. > We can introduce a new global conf to disable caching for all FileSystem, the > default would be false (or do not disable cache gobally). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17214) Allow file system caching to be disabled for all file systems
[ https://issues.apache.org/jira/browse/HADOOP-17214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183617#comment-17183617 ] Haibo Chen commented on HADOOP-17214: - The caching inside FileSystem is based on a vanilla HashMap, where the Key is partially based on UGI. Whenever UGI.loginwithKeytab() is called, the underly UGI object changes, the previously key-value pair is left unused in the cache and new entries are continuously added to the cache. Essentially we have a memory leak situation in the cache. I don't think this subtle behavior is documented anywhere, and we have seen many FileSystem users follow this pattern where UGI.loginWIthKeytab() maybe called concurrently from multiple threads. Overtime, this leads to JVM heap being filled with leaked instances in the File System cache For most of our internal FileSystem implementations (open source ones too), it is often the case that caching is left enabled (which is the default) and we would end up discovering this memory leak only in production. Having a global flag would allow us to avoid such issues in our use cases. > Allow file system caching to be disabled for all file systems > - > > Key: HADOOP-17214 > URL: https://issues.apache.org/jira/browse/HADOOP-17214 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > > Right now, FileSystem.get(URI uri, Configuration conf) allows caching of file > systems to be disabled per scheme. > We can introduce a new global conf to disable caching for all FileSystem, the > default would be false (or do not disable cache gobally). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17214) Allow file system caching to be disabled for all file systems
[ https://issues.apache.org/jira/browse/HADOOP-17214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-17214: Summary: Allow file system caching to be disabled for all file systems (was: Allow file system cache to be disabled for all file systems) > Allow file system caching to be disabled for all file systems > - > > Key: HADOOP-17214 > URL: https://issues.apache.org/jira/browse/HADOOP-17214 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > > Right now, FileSystem.get(URI uri, Configuration conf) allows caching of file > systems to be disabled per scheme. > We can introduce a new global conf to disable caching for all FileSystem, the > default would be false (or do not disable cache gobally). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17214) Allow file system cache to be disabled for all file systems
Haibo Chen created HADOOP-17214: --- Summary: Allow file system cache to be disabled for all file systems Key: HADOOP-17214 URL: https://issues.apache.org/jira/browse/HADOOP-17214 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Haibo Chen Assignee: Haibo Chen Right now, FileSystem.get(URI uri, Configuration conf) allows caching of file systems to be disabled per scheme. We can introduce a new global conf to disable caching for all FileSystem, the default would be false (or do not disable cache gobally). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15832) Upgrade BouncyCastle
[
https://issues.apache.org/jira/browse/HADOOP-15832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644246#comment-16644246
]
Haibo Chen commented on HADOOP-15832:
-
I'm +1 pushing this in as such.
> Upgrade BouncyCastle
>
>
> Key: HADOOP-15832
> URL: https://issues.apache.org/jira/browse/HADOOP-15832
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: HADOOP-15832.001.patch
>
>
> As part of my work on YARN-6586, I noticed that we're using a very old
> version of BouncyCastle:
> {code:xml}
>
>org.bouncycastle
>bcprov-jdk16
>1.46
>test
>
> {code}
> The *-jdk16 artifacts have been discontinued and are not recommended (see
> [http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html]).
>
> In particular, the newest release, 1.46, is from {color:#FF}2011{color}!
> [https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk16]
> The currently maintained and recommended artifacts are *-jdk15on:
> [https://www.bouncycastle.org/latest_releases.html]
> They're currently on version 1.60, released only a few months ago.
> We should update BouncyCastle to the *-jdk15on artifacts and the 1.60
> release. It's currently a test-only artifact, so there should be no
> backwards-compatibility issues with updating this. It's also needed for
> YARN-6586, where we'll actually be shipping it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15832) Upgrade BouncyCastle
[
https://issues.apache.org/jira/browse/HADOOP-15832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644135#comment-16644135
]
Haibo Chen commented on HADOOP-15832:
-
I manually kicked off a build at
[https://builds.apache.org/job/PreCommit-HADOOP-Build/15329/.|https://builds.apache.org/job/PreCommit-HADOOP-Build/15329/]
But looks like it is getting the same issue again.
I tried to mvn install -U locally, but could not reproduce either.
> Upgrade BouncyCastle
>
>
> Key: HADOOP-15832
> URL: https://issues.apache.org/jira/browse/HADOOP-15832
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: HADOOP-15832.001.patch
>
>
> As part of my work on YARN-6586, I noticed that we're using a very old
> version of BouncyCastle:
> {code:xml}
>
>org.bouncycastle
>bcprov-jdk16
>1.46
>test
>
> {code}
> The *-jdk16 artifacts have been discontinued and are not recommended (see
> [http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html]).
>
> In particular, the newest release, 1.46, is from {color:#FF}2011{color}!
> [https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk16]
> The currently maintained and recommended artifacts are *-jdk15on:
> [https://www.bouncycastle.org/latest_releases.html]
> They're currently on version 1.60, released only a few months ago.
> We should update BouncyCastle to the *-jdk15on artifacts and the 1.60
> release. It's currently a test-only artifact, so there should be no
> backwards-compatibility issues with updating this. It's also needed for
> YARN-6586, where we'll actually be shipping it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564419#comment-16564419
]
Haibo Chen commented on HADOOP-15644:
-
Thanks [~rkanter]
> Hadoop Docker Image Pip Install Fails on branch-2
> -
>
> Key: HADOOP-15644
> URL: https://issues.apache.org/jira/browse/HADOOP-15644
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.10.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Critical
> Fix For: 2.10.0
>
> Attachments: HADOOP-15644-branch-2.00.patch
>
>
> HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
> still failing on branch-2, though with a slightly different error message
> {code:java}
> Downloading/unpacking pylint
> Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
> package pylint
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239,
> in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264,
> in fetch_build_eggs
> replace_conflicting=True
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
> resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
> best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
> obtain
> return installer(requirement)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314,
> in fetch_build_egg
> return cmd.easy_install(req)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 616, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 646, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 834, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1040, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1025, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
> in run_setup
> lambda: execfile(
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line
> 100, in run
> return func()
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
> in
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 76, in
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243,
> in __init__
> _Distribution.__init__(self,attrs)
> File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277,
> in finalize_options
> ep.load()(self, ep.name, value)
> File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
> 10, in version_keyword
> File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
> in _warn_if_setuptools_outdated
> setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
> old (<12)
> Complete output from command python setup.py egg_info:
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174,
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated HADOOP-15644:
Description:
HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
still failing on branch-2, though with a slightly different error message
{code:java}
Downloading/unpacking pylint
Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
package pylint
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution
option: 'python_requires'
warnings.warn(msg)
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip_build_root/pylint/setup.py", line 177, in
install()
File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
**kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in
__init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in
fetch_build_eggs
replace_conflicting=True
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
obtain
return installer(requirement)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314, in
fetch_build_egg
return cmd.easy_install(req)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
616, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
646, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
834, in install_eggs
return self.build_and_install(setup_script, setup_base)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
1040, in build_and_install
self.run_setup(setup_script, setup_base, args)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
1025, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
in run_setup
lambda: execfile(
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 100,
in run
return func()
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
in
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 76, in
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243, in
__init__
_Distribution.__init__(self,attrs)
File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
self.finalize_options()
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277, in
finalize_options
ep.load()(self, ep.name, value)
File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
10, in version_keyword
File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
in _warn_if_setuptools_outdated
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
old (<12)
Complete output from command python setup.py egg_info:
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution
option: 'python_requires'
warnings.warn(msg)
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip_build_root/pylint/setup.py", line 177, in
install()
File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
**kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in
__init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in
fetch_build_eggs
replace_conflicting=True
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
best_match
return self.obtain(req, installer) # try and download/install
File
[jira] [Commented] (HADOOP-15610) Hadoop Docker Image Pip Install Fails
[ https://issues.apache.org/jira/browse/HADOOP-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564131#comment-16564131 ] Haibo Chen commented on HADOOP-15610: - I created HADOOP-15644 to fix branch-2 > Hadoop Docker Image Pip Install Fails > - > > Key: HADOOP-15610 > URL: https://issues.apache.org/jira/browse/HADOOP-15610 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Critical > Labels: docker, trunk > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15610.001.patch, HADOOP-15610.002.patch, > HADOOP-15610.003.patch, HADOOP-15610.004.patch, HADOOP-15610.005.patch > > > The Hadoop Docker image on trunk does not build. The pip package on the > Ubuntu Xenial repo is out of date and fails by throwing the following error > when attempting to install pylint: > "You are using pip version 8.1.1, however version 10.0.1 is available" > The following patch fixes this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
Haibo Chen created HADOOP-15644: --- Summary: Hadoop Docker Image Pip Install Fails on branch-2 Key: HADOOP-15644 URL: https://issues.apache.org/jira/browse/HADOOP-15644 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.10.0 Reporter: Haibo Chen Assignee: Haibo Chen Attachments: HADOOP-15644-branch-2.00.patch HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is still failing on branch-2, though with a slightly different error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15644: Attachment: HADOOP-15644-branch-2.00.patch > Hadoop Docker Image Pip Install Fails on branch-2 > - > > Key: HADOOP-15644 > URL: https://issues.apache.org/jira/browse/HADOOP-15644 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.10.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Critical > Attachments: HADOOP-15644-branch-2.00.patch > > > HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is > still failing on branch-2, though with a slightly different error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15644: Status: Patch Available (was: Open) > Hadoop Docker Image Pip Install Fails on branch-2 > - > > Key: HADOOP-15644 > URL: https://issues.apache.org/jira/browse/HADOOP-15644 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.10.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Critical > Attachments: HADOOP-15644-branch-2.00.patch > > > HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is > still failing on branch-2, though with a slightly different error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15610) Hadoop Docker Image Pip Install Fails
[
https://issues.apache.org/jira/browse/HADOOP-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564113#comment-16564113
]
Haibo Chen commented on HADOOP-15610:
-
I tried to backport this to branch-2
{code:java}
diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
index b1fc420..026109f 100644
--- a/dev-support/docker/Dockerfile
+++ b/dev-support/docker/Dockerfile
@@ -122,7 +122,7 @@ RUN apt-get -q install --no-install-recommends -y bats
# Install pylint
-RUN pip install pylint
+RUN pip install pylint==1.9.2{code}
It fixes the build failure that I ran across while running ./start-build-env.sh
> Hadoop Docker Image Pip Install Fails
> -
>
> Key: HADOOP-15610
> URL: https://issues.apache.org/jira/browse/HADOOP-15610
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Critical
> Labels: docker, trunk
> Fix For: 3.2.0, 3.1.1, 3.0.4
>
> Attachments: HADOOP-15610.001.patch, HADOOP-15610.002.patch,
> HADOOP-15610.003.patch, HADOOP-15610.004.patch, HADOOP-15610.005.patch
>
>
> The Hadoop Docker image on trunk does not build. The pip package on the
> Ubuntu Xenial repo is out of date and fails by throwing the following error
> when attempting to install pylint:
> "You are using pip version 8.1.1, however version 10.0.1 is available"
> The following patch fixes this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15610) Hadoop Docker Image Pip Install Fails
[ https://issues.apache.org/jira/browse/HADOOP-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562255#comment-16562255 ] Haibo Chen commented on HADOOP-15610: - I see. Thanks for pointing that out, [~aw]. Looks like branch-2 build is failing for a slight different issue (https://builds.apache.org/job/PreCommit-YARN-Build/21436/console) > Hadoop Docker Image Pip Install Fails > - > > Key: HADOOP-15610 > URL: https://issues.apache.org/jira/browse/HADOOP-15610 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Critical > Labels: docker, trunk > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15610.001.patch, HADOOP-15610.002.patch, > HADOOP-15610.003.patch, HADOOP-15610.004.patch, HADOOP-15610.005.patch > > > The Hadoop Docker image on trunk does not build. The pip package on the > Ubuntu Xenial repo is out of date and fails by throwing the following error > when attempting to install pylint: > "You are using pip version 8.1.1, however version 10.0.1 is available" > The following patch fixes this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15610) Hadoop Docker Image Pip Install Fails
[ https://issues.apache.org/jira/browse/HADOOP-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560445#comment-16560445 ] Haibo Chen commented on HADOOP-15610: - This is now failing with a slightly different version number "You are using pip version 8.1.1, however version 18.0 is available." both on my laptop and https://builds.apache.org/job/PreCommit-YARN-Build/21373/console > Hadoop Docker Image Pip Install Fails > - > > Key: HADOOP-15610 > URL: https://issues.apache.org/jira/browse/HADOOP-15610 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Critical > Labels: docker, trunk > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15610.001.patch, HADOOP-15610.002.patch, > HADOOP-15610.003.patch, HADOOP-15610.004.patch, HADOOP-15610.005.patch > > > The Hadoop Docker image on trunk does not build. The pip package on the > Ubuntu Xenial repo is out of date and fails by throwing the following error > when attempting to install pylint: > "You are using pip version 8.1.1, however version 10.0.1 is available" > The following patch fixes this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15586) Fix wrong log statement in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated HADOOP-15586:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~snemeth] for the fix and [~bsteinbach] for the additional review. I
have committed the patch to trunk.
> Fix wrong log statement in AbstractService
> --
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15586) Fix wrong log statement in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated HADOOP-15586:
Summary: Fix wrong log statement in AbstractService (was: Fix wrong log
statements in AbstractService)
> Fix wrong log statement in AbstractService
> --
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15586) Fix wrong log statements in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551065#comment-16551065
]
Haibo Chen commented on HADOOP-15586:
-
{quote}LOG.debug has one signature to use a format string plus the objects as
arguments to format, and the other one is String plus a Throwable instance
{quote}
I see.
I will check it in later today if no objections.
I would revert the change to the indention when I commit to be consistent with
the rest of the code base which uses 4 spaces.
> Fix wrong log statements in AbstractService
> ---
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15586) Fix wrong log statements in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550025#comment-16550025
]
Haibo Chen commented on HADOOP-15586:
-
Hi [~snemeth]. I'd echo [~ste...@apache.org]'s previous comment, we just need
to change "LOG.debug("noteFailure {}" + exception);" to "LOG.debug("noteFailure
{}", exception);"
The other change is unnecessary, and "Exception in noteFailure" is a bit
misleading because it is not an exception thrown/caught in noteFailure().
noteFailure() is a just helper method.
> Fix wrong log statements in AbstractService
> ---
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15507: Attachment: (was: YARN-8240-YARN-1011.02.patch) > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15507: Attachment: YARN-8240-YARN-1011.02.patch > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, YARN-8240-YARN-1011.02.patch, > image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500612#comment-16500612 ] Haibo Chen commented on HADOOP-15507: - The MapReduce part looks good to me, but I'm not very familiar with the erase code or the file system API to comment on the rest of the patch. > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213736#comment-16213736 ] Haibo Chen commented on HADOOP-14957: - Given I don't have any idea of the scope of the potential impact, unassign myself here. Feel free to take it over. > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14957.prelim.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned HADOOP-14957: --- Assignee: (was: Haibo Chen) > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen > Attachments: HADOOP-14957.prelim.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14957: Status: Patch Available (was: Open) > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14957.prelim.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213259#comment-16213259 ] Haibo Chen commented on HADOOP-14957: - Any client that's not directly calling the ClientDatanodeProtocol method in question should be fine given the .proto data is unchanged as you point out. Looks like it is just used in DFS Admin protocol, not sure how pervasive it is used. [~xiaochen] [~andrew.wang]. I have attached a patch that I have tested internally to make sure it does not cause build failures for downstream projects, in case we decide to change it. > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14957.prelim.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14957: Hadoop Flags: Incompatible change > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14957.prelim.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14957: Attachment: HADOOP-14957.prelim.patch > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14957.prelim.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14771: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch, > HADOOP-14771.03.patch, HADOOP-14771.04.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211479#comment-16211479 ] Haibo Chen commented on HADOOP-14771: - I ran mvn install locally and some internal tests that make sure downstream projects build successfully. +1 on the v4 patch. Will commit it later today if there is no objection > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch, > HADOOP-14771.03.patch, HADOOP-14771.04.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14043) Shade netty 4 dependency in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HADOOP-14043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved HADOOP-14043. - Resolution: Won't Fix > Shade netty 4 dependency in hadoop-hdfs > --- > > Key: HADOOP-14043 > URL: https://issues.apache.org/jira/browse/HADOOP-14043 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Ted Yu >Priority: Critical > > During review of HADOOP-13866, [~andrew.wang] mentioned considering shading > netty before putting the fix into branch-2. > This would give users better experience when upgrading hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14043) Shade netty 4 dependency in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HADOOP-14043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210264#comment-16210264 ] Haibo Chen commented on HADOOP-14043: - Closing this since we won't fix HADOOP-13866. Feel free to reopen if you disagree. > Shade netty 4 dependency in hadoop-hdfs > --- > > Key: HADOOP-14043 > URL: https://issues.apache.org/jira/browse/HADOOP-14043 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Ted Yu >Priority: Critical > > During review of HADOOP-13866, [~andrew.wang] mentioned considering shading > netty before putting the fix into branch-2. > This would give users better experience when upgrading hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208468#comment-16208468 ] Haibo Chen commented on HADOOP-14238: - +1 on hooking it into Yetus if it is not a lot of work. > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Critical > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14043) Shade netty 4 dependency in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HADOOP-14043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207890#comment-16207890 ] Haibo Chen commented on HADOOP-14043: - [~tedyu], are we shading netty solely because we want to backport HADOOP-13866 to branch-2? I am revisiting HADOOP-13866 and thinking that it is no longer needed, see my comments there. If HADOOP-13866 is a won't fix, do we still need to shade netty? > Shade netty 4 dependency in hadoop-hdfs > --- > > Key: HADOOP-14043 > URL: https://issues.apache.org/jira/browse/HADOOP-14043 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Ted Yu >Priority: Critical > > During review of HADOOP-13866, [~andrew.wang] mentioned considering shading > netty before putting the fix into branch-2. > This would give users better experience when upgrading hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207884#comment-16207884
]
Haibo Chen commented on HADOOP-13866:
-
To summarize the current status, the netty-all version has been reverted back
to a 4.0 stable release in both trunk and branch-2,
and HBases shades its netty dependency so that it no longer clashes with the
version of netty hadoop depends. Thus, there is
no need to upgrade netty-all to 4.1.1.Final any more.
I am closing this as won't fix unless there is any objection.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch, HADOOP-13866.v9.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
[ https://issues.apache.org/jira/browse/HADOOP-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14957: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14238 > ReconfigurationTaskStatus is exposing guava Optional in its public api > -- > > Key: HADOOP-14957 > URL: https://issues.apache.org/jira/browse/HADOOP-14957 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14957) ReconfigurationTaskStatus is exposing guava Optional in its public api
Haibo Chen created HADOOP-14957: --- Summary: ReconfigurationTaskStatus is exposing guava Optional in its public api Key: HADOOP-14957 URL: https://issues.apache.org/jira/browse/HADOOP-14957 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.0.0-beta1 Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206808#comment-16206808 ] Haibo Chen commented on HADOOP-14238: - Manually checking hadoop-yarn-api, hadoop-yarn-applications, hadoop-yarn-common, hadoop-yarn-client and hadoop-yarn-registry, there is no exposure either. We do, however, expose Optional in ReconfigurationTaskStatus which is marked as public and stable. I think this needs to be fixed. It'll still be great if we can still get the tool to double check just in case, cc [~andrew.wang]. > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Critical > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206781#comment-16206781 ] Haibo Chen commented on HADOOP-14238: - I checked mapreduce client modules manually, that is, hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-core, hadoop-mapreduce-client-common. There is no exposure of GUAVA in the public api. > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Critical > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12436) GlobPattern regex library has performance issues with wildcard characters
[
https://issues.apache.org/jira/browse/HADOOP-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated HADOOP-12436:
Release Note: GlobFilter and RegexFilter.compile() now returns
com.google.re2j.pattern.Pattern instead of java.util.regex.Pattern
> GlobPattern regex library has performance issues with wildcard characters
> -
>
> Key: HADOOP-12436
> URL: https://issues.apache.org/jira/browse/HADOOP-12436
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.2.0, 2.7.1
>Reporter: Matthew Paduano
>Assignee: Matthew Paduano
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12436.01.patch, HADOOP-12436.02.patch,
> HADOOP-12436.03.patch, HADOOP-12436.04.patch, HADOOP-12436.05.patch
>
>
> java.util.regex classes have performance problems with certain wildcard
> patterns. Namely, consecutive * characters in a file name (not properly
> escaped as literals) will cause commands such as "hadoop fs -ls
> file**name" to consume 100% CPU and probably never return in a reasonable
> time (time scales with number of *'s).
> Here is an example:
> {noformat}
> hadoop fs -touchz
> /user/mattp/job_1429571161900_4222-1430338332599-tda%2D%2D\\\+\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\+\\\+\\\+...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist
> hadoop fs -ls
> /user/mattp/job_1429571161900_4222-1430338332599-tda%2D%2D+**+++...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist
> {noformat}
> causes:
> {noformat}
> PIDCOMMAND %CPU TIME
> 14526 java 100.0 01:18.85
> {noformat}
> Not every string of *'s causes this, but the above filename reproduces this
> reliably.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12436) GlobPattern regex library has performance issues with wildcard characters
[
https://issues.apache.org/jira/browse/HADOOP-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199434#comment-16199434
]
Haibo Chen commented on HADOOP-12436:
-
[~aw] [~mattpaduano] This seems an incompatible change given GlobFilter and
RegexFilter are Public Evolving. Hence, I have added an incompatible tag. Feel
free to remove it if you disagree
> GlobPattern regex library has performance issues with wildcard characters
> -
>
> Key: HADOOP-12436
> URL: https://issues.apache.org/jira/browse/HADOOP-12436
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.2.0, 2.7.1
>Reporter: Matthew Paduano
>Assignee: Matthew Paduano
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12436.01.patch, HADOOP-12436.02.patch,
> HADOOP-12436.03.patch, HADOOP-12436.04.patch, HADOOP-12436.05.patch
>
>
> java.util.regex classes have performance problems with certain wildcard
> patterns. Namely, consecutive * characters in a file name (not properly
> escaped as literals) will cause commands such as "hadoop fs -ls
> file**name" to consume 100% CPU and probably never return in a reasonable
> time (time scales with number of *'s).
> Here is an example:
> {noformat}
> hadoop fs -touchz
> /user/mattp/job_1429571161900_4222-1430338332599-tda%2D%2D\\\+\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\+\\\+\\\+...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist
> hadoop fs -ls
> /user/mattp/job_1429571161900_4222-1430338332599-tda%2D%2D+**+++...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist
> {noformat}
> causes:
> {noformat}
> PIDCOMMAND %CPU TIME
> 14526 java 100.0 01:18.85
> {noformat}
> Not every string of *'s causes this, but the above filename reproduces this
> reliably.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12436) GlobPattern regex library has performance issues with wildcard characters
[
https://issues.apache.org/jira/browse/HADOOP-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated HADOOP-12436:
Target Version/s: (was: )
Hadoop Flags: Incompatible change
> GlobPattern regex library has performance issues with wildcard characters
> -
>
> Key: HADOOP-12436
> URL: https://issues.apache.org/jira/browse/HADOOP-12436
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.2.0, 2.7.1
>Reporter: Matthew Paduano
>Assignee: Matthew Paduano
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12436.01.patch, HADOOP-12436.02.patch,
> HADOOP-12436.03.patch, HADOOP-12436.04.patch, HADOOP-12436.05.patch
>
>
> java.util.regex classes have performance problems with certain wildcard
> patterns. Namely, consecutive * characters in a file name (not properly
> escaped as literals) will cause commands such as "hadoop fs -ls
> file**name" to consume 100% CPU and probably never return in a reasonable
> time (time scales with number of *'s).
> Here is an example:
> {noformat}
> hadoop fs -touchz
> /user/mattp/job_1429571161900_4222-1430338332599-tda%2D%2D\\\+\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\+\\\+\\\+...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist
> hadoop fs -ls
> /user/mattp/job_1429571161900_4222-1430338332599-tda%2D%2D+**+++...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist
> {noformat}
> causes:
> {noformat}
> PIDCOMMAND %CPU TIME
> 14526 java 100.0 01:18.85
> {noformat}
> Not every string of *'s causes this, but the above filename reproduces this
> reliably.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197171#comment-16197171 ] Haibo Chen commented on HADOOP-14771: - Sorry for losing this on my radar, [~ajayydv]. I'll take a look at the latest patch some time this week. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch, > HADOOP-14771.03.patch, HADOOP-14771.04.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171890#comment-16171890 ] Haibo Chen commented on HADOOP-14771: - Sorry for the breakage of trunk, folks. I will be more careful with future commits. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170953#comment-16170953 ] Haibo Chen commented on HADOOP-14771: - I reverted HADOOP-14771 locally and ran mvn clean install -DskipTests, still see the error show up. Am I missing something? [~busbey] As the author of the check script, can you provide some insight of this? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170893#comment-16170893 ] Haibo Chen commented on HADOOP-14771: - Never mind. See it in the link above > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170892#comment-16170892 ] Haibo Chen commented on HADOOP-14771: - [~asuresh] can you point me to the build failure? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170754#comment-16170754 ] Haibo Chen commented on HADOOP-14771: - Thanks [~ajayydv] for the contribution, [~bharatviswa] and [~busbey] for the guidance. I have committed it to branch-3.0 and trunk. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14771: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 Status: Resolved (was: Patch Available) > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170168#comment-16170168 ] Haibo Chen commented on HADOOP-14771: - bq. I think we can include some transitive dependencies we are sure of right now. And update it incrementally (i.e exclusion list) in future Agreed. +1 on the latest patch. Will commit it at the end of today unless there are further comments. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14871) Hadoop RunJar needs more debug logs.
[ https://issues.apache.org/jira/browse/HADOOP-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170156#comment-16170156 ] Haibo Chen commented on HADOOP-14871: - Yes, [~bharatviswa]. I am referring to the client side class loader that is configurable in RunJar class. > Hadoop RunJar needs more debug logs. > > > Key: HADOOP-14871 > URL: https://issues.apache.org/jira/browse/HADOOP-14871 > Project: Hadoop Common > Issue Type: Wish >Reporter: Haibo Chen >Priority: Minor > Labels: newbie++ > > Hadoop RunJar does not have much logging. In one case, one user had specified > MAIN class in the jar and also provided the main class as one of the > following arguments. > The extra argument to specify the main class was passed to the main class as > an argument, which caused confusing exceptions. It'd be nice to have some > debug logging > so that users can be warned. > In another case, we could have some logging for the client classloader as > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168712#comment-16168712 ] Haibo Chen commented on HADOOP-14771: - Also, it'd be nice if your can add a comment to explain why hadoop-yarn-api is excluded. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168708#comment-16168708 ] Haibo Chen commented on HADOOP-14771: - Thanks [~ajayydv] for the explanation! Excluding hadoop-yarn-api makes sense to me. Exclusion of hadoop-yarn-common seems unnecessary though (If we were to follow the same practice, we'd need to exclude a lot more transitive dependencies in other dependencies). I have manually kicked off the jenkins job to see if it breaks anything. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168709#comment-16168709 ] Haibo Chen commented on HADOOP-14771: - Oops, the patch no longer applies. Do you mind uploading a rebased version of your patch? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module
[ https://issues.apache.org/jira/browse/HADOOP-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168515#comment-16168515 ] Haibo Chen commented on HADOOP-14853: - Thanks @Robert Kanter for the review > hadoop-mapreduce-client-app is not a client module > -- > > Key: HADOOP-14853 > URL: https://issues.apache.org/jira/browse/HADOOP-14853 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14853.00.patch > > > hadoop-mapreduce-client-app is not a client module, and thus can be removed > as a dependency from hadoop-client module -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14871) Hadoop RunJar needs more debug logs.
[ https://issues.apache.org/jira/browse/HADOOP-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14871: Description: Hadoop RunJar does not have much logging. In one case, one user had specified MAIN class in the jar and also provided the main class as one of the following arguments. The extra argument to specify the main class was passed to the main class as an argument, which caused confusing exceptions. It'd be nice to have some debug logging so that users can be warned. In another case, we could have some logging for the client classloader as well. > Hadoop RunJar needs more debug logs. > > > Key: HADOOP-14871 > URL: https://issues.apache.org/jira/browse/HADOOP-14871 > Project: Hadoop Common > Issue Type: Wish >Reporter: Haibo Chen >Priority: Minor > Labels: newbie++ > > Hadoop RunJar does not have much logging. In one case, one user had specified > MAIN class in the jar and also provided the main class as one of the > following arguments. > The extra argument to specify the main class was passed to the main class as > an argument, which caused confusing exceptions. It'd be nice to have some > debug logging > so that users can be warned. > In another case, we could have some logging for the client classloader as > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14871) Hadoop RunJar needs more debug logs.
Haibo Chen created HADOOP-14871: --- Summary: Hadoop RunJar needs more debug logs. Key: HADOOP-14871 URL: https://issues.apache.org/jira/browse/HADOOP-14871 Project: Hadoop Common Issue Type: Wish Reporter: Haibo Chen Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module
[ https://issues.apache.org/jira/browse/HADOOP-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168443#comment-16168443 ] Haibo Chen commented on HADOOP-14853: - manually triggered the jenkins job. > hadoop-mapreduce-client-app is not a client module > -- > > Key: HADOOP-14853 > URL: https://issues.apache.org/jira/browse/HADOOP-14853 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14853.00.patch > > > hadoop-mapreduce-client-app is not a client module, and thus can be removed > as a dependency from hadoop-client module -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167107#comment-16167107 ] Haibo Chen commented on HADOOP-14771: - dependency:analyze gives a list of used undeclared dependencies and another list of unused declared dependencies, which we can use to ensure a good hygiene of project dependencies. IIUC, we'll make changes in hadoop-yarn-client pom.xml directly, which is different from what we are doing here, that is, excluding hadoop-yarn-client's transitive dependencies in hadoop-client. I'll try to go over the classes manually. Looking forwarding to your new tool to address such issue. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166730#comment-16166730 ] Haibo Chen commented on HADOOP-14771: - [~ajayydv] Can you shed some light on why you excluded hadoop-yarn-api and hadoop-yarn-common? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166727#comment-16166727 ] Haibo Chen commented on HADOOP-14771: - Thanks [~busbey] for the response. bq. though we end up needing it because of other things Some of the dependencies seem obvious whether they are needed or not, but others are not clear to me. Do you know how we can reliable check if one is needed and by what, other than going over the classes that downstream projects are expected to use? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161535#comment-16161535 ] Haibo Chen commented on HADOOP-14771: - Thanks [~ajayydv] for the patch! One question for you, and maybe [~busbey] can answer too. Why are excluding commons-cli, log4j, hadoop-yarn-api and hadoop-yarn-common? We don't exclude log4j and commons-cli in hadoop-common dependency, so we should probably do it consistently? hadoop-yarn-api and hadoop-yarn-common are needed by clients, no ? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14771: Priority: Critical (was: Blocker) > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161498#comment-16161498 ] Haibo Chen commented on HADOOP-14771: - bq. this hadoop-client is from hadoop 2.x it is not new right? No, it is not new. I still think the fix is required because the transitive dependency on hadoop-yarn-client is not obvious. But given nothing really breaks, I'll downgrade this to very important. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14847) Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc
[ https://issues.apache.org/jira/browse/HADOOP-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159454#comment-16159454 ] Haibo Chen commented on HADOOP-14847: - Thanks for the reminder, [~vinodkv]. Will do now. > Remove Guava Supplier and change to java Supplier in AMRMClient and > AMRMClientAysnc > --- > > Key: HADOOP-14847 > URL: https://issues.apache.org/jira/browse/HADOOP-14847 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14847.patch > > > Remove the Guava library Supplier usage in user facing API's in > AMRMClient.java and AMRMClientAsync.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14847) Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc
[ https://issues.apache.org/jira/browse/HADOOP-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14847: Fix Version/s: 3.0.0-beta1 > Remove Guava Supplier and change to java Supplier in AMRMClient and > AMRMClientAysnc > --- > > Key: HADOOP-14847 > URL: https://issues.apache.org/jira/browse/HADOOP-14847 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14847.patch > > > Remove the Guava library Supplier usage in user facing API's in > AMRMClient.java and AMRMClientAsync.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14847) Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc
[ https://issues.apache.org/jira/browse/HADOOP-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14847: Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Status: Resolved (was: Patch Available) Thanks [~bharatviswa] for your contribution. I have committed it to trunk and branch-3.0 > Remove Guava Supplier and change to java Supplier in AMRMClient and > AMRMClientAysnc > --- > > Key: HADOOP-14847 > URL: https://issues.apache.org/jira/browse/HADOOP-14847 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Blocker > Attachments: HADOOP-14847.patch > > > Remove the Guava library Supplier usage in user facing API's in > AMRMClient.java and AMRMClientAsync.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module
[ https://issues.apache.org/jira/browse/HADOOP-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14853: Status: Patch Available (was: Open) > hadoop-mapreduce-client-app is not a client module > -- > > Key: HADOOP-14853 > URL: https://issues.apache.org/jira/browse/HADOOP-14853 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14853.00.patch > > > hadoop-mapreduce-client-app is not a client module, and thus can be removed > as a dependency from hadoop-client module -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module
[ https://issues.apache.org/jira/browse/HADOOP-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14853: Attachment: HADOOP-14853.00.patch > hadoop-mapreduce-client-app is not a client module > -- > > Key: HADOOP-14853 > URL: https://issues.apache.org/jira/browse/HADOOP-14853 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-14853.00.patch > > > hadoop-mapreduce-client-app is not a client module, and thus can be removed > as a dependency from hadoop-client module -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module
Haibo Chen created HADOOP-14853: --- Summary: hadoop-mapreduce-client-app is not a client module Key: HADOOP-14853 URL: https://issues.apache.org/jira/browse/HADOOP-14853 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0-alpha4 Reporter: Haibo Chen Assignee: Haibo Chen hadoop-mapreduce-client-app is not a client module, and thus can be removed as a dependency from hadoop-client module -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158820#comment-16158820 ] Haibo Chen commented on HADOOP-14771: - Thanks [~bharatviswa] for the info! In that case, this may not be a blocker but definitely still very important to fix, as we probably want to be explicit about this. What's your thoughts, [~busbey]? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14847) Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc
[ https://issues.apache.org/jira/browse/HADOOP-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157921#comment-16157921 ] Haibo Chen commented on HADOOP-14847: - Thanks [~bharatviswa] for the patch! LGTM +1. Will commit it tomorrow unless there is future comment. > Remove Guava Supplier and change to java Supplier in AMRMClient and > AMRMClientAysnc > --- > > Key: HADOOP-14847 > URL: https://issues.apache.org/jira/browse/HADOOP-14847 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Blocker > Attachments: HADOOP-14847.patch > > > Remove the Guava library Supplier usage in user facing API's in > AMRMClient.java and AMRMClientAsync.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140720#comment-16140720
]
Haibo Chen commented on HADOOP-14284:
-
Done.
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140672#comment-16140672
]
Haibo Chen commented on HADOOP-14284:
-
[~busbey], the shaded hadoop-client work seems incomplete. Can you take a look
at HADOOP-14771 and see if it is valid?
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130894#comment-16130894
]
Haibo Chen commented on HADOOP-14284:
-
That's good to know. The TimelineServiceV2 backend is designed to be pluggable,
so doing this will help other backend implementations, if any. Regardless, if
we choose to shade server modules in addition to client modules, we don't need
to think of whether we are in a client or server module because of the
consistency and implementing the dependency enforcer will probably be easier.
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129648#comment-16129648
]
Haibo Chen edited comment on HADOOP-14284 at 8/17/17 12:03 AM:
---
Shading the whole code base also makes more sense for hadoop IMO, given the
fact that, as [~andrew.wang] pointed out in the comment above, downstream
projects still depend on hadoop-hdfs, which is a server module. In YARN-7028
and YARN-6414, we also discovered that the guava version that hbase-client
depends on can conflict with the guava version that hadoop uses, shading server
modules (hbase-client is embedded in NM and RM) will help avoid that issue.
was (Author: haibochen):
Shading the whole code base also makes more sense for hadoop IMO, given the
fact that, as [~andrew.wang] pointed out in the comment above, downstream
projects still depend on hadoop-hdfs, which is a server module. In YARN-7028,
we also discovered that the guava version that hbase-client depends on can
conflict with the guava version that hadoop uses, shading server modules
(hbase-client is embedded in NM and RM) will help avoid that issue.
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129648#comment-16129648
]
Haibo Chen commented on HADOOP-14284:
-
Shading the whole code base also makes more sense for hadoop IMO, given the
fact that, as [~andrew.wang] pointed out in the comment above, downstream
projects still depend on hadoop-hdfs, which is a server module. In YARN-7028,
we also discovered that the guava version that hbase-client depends on can
conflict with the guava version that hadoop uses, shading server modules
(hbase-client is embedded in NM and RM) will help avoid that issue.
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14771: Summary: hadoop-client does not include hadoop-yarn-client (was: hadoop-common does not include hadoop-yarn-client) > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Priority: Critical > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126259#comment-16126259
]
Haibo Chen commented on HADOOP-14284:
-
Thanks @stack for the insight on HBase's solution. Does HBase shade across all
code base, instead of just in client modules?
bq. Downside is having to be sure we always refer to the relocated versions in
code.
Seems like maven enforcer can ban the unwanted non-relocated dependencies,
manageable if provided that we are willing to reference the relocated versions
in both server and client modules. If this does work, It seems to me that
shading both server and client modules would be beneficial in the long run from
maintenance point of view, even though what downstream projects need is only
shaded clients. Thoughts?
Does this approach address all of your concerns, [~djp]?
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14771) hadoop-common does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14771: Issue Type: Sub-task (was: Bug) Parent: HADOOP-11656 > hadoop-common does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Priority: Critical > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-common does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126071#comment-16126071 ] Haibo Chen commented on HADOOP-14771: - [~busbey] Do you think this is a blocker for 3.0 beta1? HADOOP-11656 (Classpath isolation for downstream clients) is a blocker, so I think this one should be too, but I'll let you to determine. > hadoop-common does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Haibo Chen >Priority: Critical > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14771) hadoop-common does not include hadoop-yarn-client
Haibo Chen created HADOOP-14771: --- Summary: hadoop-common does not include hadoop-yarn-client Key: HADOOP-14771 URL: https://issues.apache.org/jira/browse/HADOOP-14771 Project: Hadoop Common Issue Type: Bug Components: common Reporter: Haibo Chen Priority: Critical The hadoop-client does not include hadoop-yarn-client, thus, the shared hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119162#comment-16119162
]
Haibo Chen commented on HADOOP-14284:
-
Following through the discussions, folks seem to prefer shading just client
jars since we still want to use/upgrade guava in hadoop and we do not want to
enforce the guava version on downstream projects.
[~djp] Can you elaborate on why you think the shaded yarn/mr client is blocking
the work here? Cann't we do them in parallel?
To reduce build time, we could move the third-party module into an auxiliary
hadoop project, like what HBase community is doing, so that we avoid shading
every time hadoop is built. Thoughts?
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch,
> HADOOP-14284.012.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-14409) Yarn and HDFS configuration should use SSLFactory.class
[ https://issues.apache.org/jira/browse/HADOOP-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen moved YARN-6579 to HADOOP-14409: --- Component/s: (was: yarn) (was: security) (was: webapp) security Key: HADOOP-14409 (was: YARN-6579) Project: Hadoop Common (was: Hadoop YARN) > Yarn and HDFS configuration should use SSLFactory.class > --- > > Key: HADOOP-14409 > URL: https://issues.apache.org/jira/browse/HADOOP-14409 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Vlad Rozov > > There are multiple configuration parameters provided by SSLFactory that are > duplicated by YarnConfiguration and other classes. It will be good to unify > how Hadoop SSL endpoints are configured by using SSLFactory. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14409) Yarn and HDFS should reuse SSLFactory configurations
[ https://issues.apache.org/jira/browse/HADOOP-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-14409: Summary: Yarn and HDFS should reuse SSLFactory configurations (was: Yarn and HDFS configuration should use SSLFactory.class) > Yarn and HDFS should reuse SSLFactory configurations > > > Key: HADOOP-14409 > URL: https://issues.apache.org/jira/browse/HADOOP-14409 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Vlad Rozov > > There are multiple configuration parameters provided by SSLFactory that are > duplicated by YarnConfiguration and other classes. It will be good to unify > how Hadoop SSL endpoints are configured by using SSLFactory. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975258#comment-15975258
]
Haibo Chen commented on HADOOP-13866:
-
Thanks for the pointer [~tedyu]
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch, HADOOP-13866.v9.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971427#comment-15971427
]
Haibo Chen commented on HADOOP-13866:
-
Reading through the discussion above, the decision is to go back to
4.0.23.Final (HDFS-11498) and have HBase exclude the netty-all dependency from
HADOOP. [~tedyu], is there a HBASE jira to exclude the HADOOP dependency?
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch, HADOOP-13866.v9.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971427#comment-15971427
]
Haibo Chen edited comment on HADOOP-13866 at 4/17/17 6:09 PM:
--
Reading through the discussion above, the decision seems to be going back to
4.0.23.Final (HDFS-11498) and having HBase exclude the netty-all dependency
from HADOOP. [~tedyu], is there a HBASE jira to exclude the HADOOP dependency?
was (Author: haibochen):
Reading through the discussion above, the decision is to go back to
4.0.23.Final (HDFS-11498) and have HBase exclude the netty-all dependency from
HADOOP. [~tedyu], is there a HBASE jira to exclude the HADOOP dependency?
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch, HADOOP-13866.v9.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[
https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966877#comment-15966877
]
Haibo Chen commented on HADOOP-14300:
-
Thanks @[~damien.claveau] for reporting this and providing a patch! Can you
also add a unit test in TestGraphiteSink?
> GraphiteSink reports metrics containing undesirable whitespaces
> ---
>
> Key: HADOOP-14300
> URL: https://issues.apache.org/jira/browse/HADOOP-14300
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3
>Reporter: Damien Claveau
>Priority: Trivial
> Attachments: HADOOP-14300
>
>
> According to the Graphite documentation, the data sent must be in the
> following format:
> But some tag values that are flattened in the metricpath string sometimes
> include whitespaces and violate the carbon/graphite format.
> For example, the dirpath enumeration from the datanode :
> FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current,
> /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609
> Currently, none of the aggregators available in the Graphite ecosystem can
> handle these malformed metrics. These whitespaces should be replaced by
> underscores.
> I will be happy to attach a small patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14218) Replace assertThat with assertTrue in MetricsAsserts
[
https://issues.apache.org/jira/browse/HADOOP-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943530#comment-15943530
]
Haibo Chen commented on HADOOP-14218:
-
+1 nonbinding
> Replace assertThat with assertTrue in MetricsAsserts
>
>
> Key: HADOOP-14218
> URL: https://issues.apache.org/jira/browse/HADOOP-14218
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HADOOP-14218.01.patch
>
>
> {code:title=MetricsAsserts.java}
> public static void assertCounterGt(String name, long greater,
> MetricsRecordBuilder rb) {
> Assert.assertThat("Bad value for metric " + name, getLongCounter(name,
> rb),
> new GreaterThan(greater));
> }
> {code}
> The following code cannot be compiled with Mockito 2.1+ because it does not
> depend on org.hamcrest.Matcher anymore. We can simply replace this code with
> {{assertTrue(message, getLongCounter() > greater)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838786#comment-15838786
]
Haibo Chen commented on HADOOP-13866:
-
See [~xiaochen]'s comment above.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838772#comment-15838772
]
Haibo Chen commented on HADOOP-13866:
-
bq. If I can construct Http2ConnectionHandler using above code,
DtpHttp2FrameListener can be plugged in which should make
TestDatanodeHttpXFrame pass.
Yes, this is what I was also thinking of. But let's wait for a while to see if
HDFS-8377 can be reverted. It will make the change much easier.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751894#comment-15751894
]
Haibo Chen commented on HADOOP-13866:
-
You can call this inside DtpHttp2FrameListener constructor, then the rest of
hadoop code can just call DtpHttp2FrameListener constructor.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749879#comment-15749879
]
Haibo Chen commented on HADOOP-13866:
-
I suspect it is this code in DtpHttp2FrameListener
{code:java}
encoder.writeHeaders(ctx, streamId,
new DefaultHttp2Headers().status(HttpResponseStatus.OK.codeAsText()), 0,
false, ctx.newPromise());
{code}
Inspecting the source code in 4.1.0.Beta5, the Http2ConnectionHandler
constructor that is called in hadoop instantiates some default implementations.
However, the constructor is removed from 4.1.1.Final. We can add code in hadoop
to instantiate default implementations and pass them to new
Http2ConnectionHandler constructor.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749702#comment-15749702
]
Haibo Chen commented on HADOOP-13866:
-
[~ted_yu] Thanks for the patch! The test failure might have something to do
with DtpHttp2Handler, which sets up DtpHttp2FrameListener (which in turns
writes some header information). I think we should not delete DtpHttp2Handler
but make changes to it.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[ https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-13866: Assignee: Ted Yu (was: Haibo Chen) > Upgrade netty-all to 4.1.1.Final > > > Key: HADOOP-13866 > URL: https://issues.apache.org/jira/browse/HADOOP-13866 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch > > > netty-all 4.1.1.Final is stable release which we should upgrade to. > See bottom of HADOOP-12927 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[ https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-13866: Assignee: Haibo Chen > Upgrade netty-all to 4.1.1.Final > > > Key: HADOOP-13866 > URL: https://issues.apache.org/jira/browse/HADOOP-13866 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Haibo Chen > Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch > > > netty-all 4.1.1.Final is stable release which we should upgrade to. > See bottom of HADOOP-12927 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[ https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723411#comment-15723411 ] Haibo Chen commented on HADOOP-13866: - My quick check shows that upgrading to 4.1.1.Final breaks DtpHttp2Handler. Looks like 4.1.1.Final removed some methods that hdfs current uses. > Upgrade netty-all to 4.1.1.Final > > > Key: HADOOP-13866 > URL: https://issues.apache.org/jira/browse/HADOOP-13866 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu > Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch > > > netty-all 4.1.1.Final is stable release which we should upgrade to. > See bottom of HADOOP-12927 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12927) Update netty-all to 4.0.34.Final
[ https://issues.apache.org/jira/browse/HADOOP-12927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723185#comment-15723185 ] Haibo Chen commented on HADOOP-12927: - [~ted_yu] I think we can close this as not applicable and create a new jira. > Update netty-all to 4.0.34.Final > > > Key: HADOOP-12927 > URL: https://issues.apache.org/jira/browse/HADOOP-12927 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan > > Pull request: https://github.com/apache/hadoop/pull/84 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13859) TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties.
[ https://issues.apache.org/jira/browse/HADOOP-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716752#comment-15716752 ] Haibo Chen commented on HADOOP-13859: - The unit test failure as well as the findbug warning looks irrelevant to me . > TestConfigurationFieldsBase fails for fields that are DEFAULT values of > skipped properties. > --- > > Key: HADOOP-13859 > URL: https://issues.apache.org/jira/browse/HADOOP-13859 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: HADOOP-13859.01.patch > > > In YARN-5922, two new default values are added in YarnConfiguration for two > timeline-service properties that are skipped in TestConfigurationFieldsBase. > TestConfigurationFieldsBase fails as it mistakenly treats the two newly added > default-values as regular properties. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
