[
https://issues.apache.org/jira/browse/TEZ-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100367#comment-17100367
]
Jonathan Turner Eagles commented on TEZ-4157:
---------------------------------------------
It can be difficult to debug Http as java does things under the hood without
logging. I have tried to debug by enabling logging in sun's http code.
Here are the instructions to enable logging.
{noformat}
$ cat tez-plugins/tez-aux-services/src/test/resources/logging.properties
handlers= java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = FINEST
sun.net.www.protocol.http.HttpURLConnection.level=ALL
{noformat}
I then instructed the IDE to enable logging using this properties file
{noformat:title=jvm}
-Djava.util.logging.config.file=/Users/jeagles/hadoop/tez/tez-plugins/tez-aux-services/src/test/resources/logging.properties
{noformat}
{noformat:title=First connect}
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
plainConnect0
FINEST: ProxySelector Request for
http://127.0.0.1:57603/mapOutput?job=job_12345_1&dag=1&reduce=1&map=attempt_12345_1_m_1_0
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
plainConnect0
FINEST: Proxy used: DIRECT
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
writeRequests
FINE: sun.net.www.MessageHeader@d737b897 pairs: {GET
/mapOutput?job=job_12345_1&dag=1&reduce=1&map=attempt_12345_1_m_1_0 HTTP/1.1:
null}{name: mapreduce}{version: 1.0.0}{User-Agent: Java/1.8.0_202}{Host:
127.0.0.1:57603}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*;
q=.2}{Connection: keep-alive}
{noformat}
{noformat:title=First response}
May 05, 2020 7:46:21 PM sun.net.www.http.HttpClient logFinest
FINEST: KeepAlive stream used:
http://127.0.0.1:57603/mapOutput?job=job_12345_1&dag=1&reduce=1&map=attempt_12345_1_m_1_0
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
getInputStream0
FINE: sun.net.www.MessageHeader@8519cb44 pairs: {null: HTTP/1.1 200
OK}{Content-Length: 2900000}{Connection: keep-alive}{keep-alive: timeout=60}
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
plainConnect0
FINEST: ProxySelector Request for
http://127.0.0.1:57603/mapOutput?job=job_12345_1&dag=1&reduce=1&map=attempt_12345_1_m_1_0&keepAlive=true
May 05, 2020 7:46:21 PM sun.net.www.http.HttpClient logFinest
FINEST: KeepAlive stream retrieved from the cache,
sun.net.www.http.HttpClient(http://127.0.0.1:57603/mapOutput?job=job_12345_1&dag=1&reduce=1&map=attempt_12345_1_m_1_0)
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
plainConnect0
FINEST: Proxy used: DIRECT
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
writeRequests
FINE: sun.net.www.MessageHeader@35dab4eb7 pairs: {GET
/mapOutput?job=job_12345_1&dag=1&reduce=1&map=attempt_12345_1_m_1_0&keepAlive=true
HTTP/1.1: null}{name: mapreduce}{version: 1.0.0}{User-Agent:
Java/1.8.0_202}{Host: 127.0.0.1:57603}{Accept: text/html, image/gif,
image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
{noformat}
{noformat:title=Second connect}
May 05, 2020 7:46:21 PM sun.net.www.protocol.http.HttpURLConnection
getInputStream0
FINE: sun.net.www.MessageHeader@2d901eb01 pairs: {Content-type: unknown/unknown}
java.io.IOException: Invalid Http response
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1606)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at
org.apache.tez.auxservices.TestShuffleHandler.testKeepAlive(TestShuffleHandler.java:461)
{noformat}
> ShuffleHandler: upgrade to netty4
> ---------------------------------
>
> Key: TEZ-4157
> URL: https://issues.apache.org/jira/browse/TEZ-4157
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4157.01.patch, TEZ-4157.02.patch
>
>
> -In the dependency tree, there are 2 occurrences of compile scope direct
> netty dependencies, however, they're not used at all. I compiled locally
> successfully without them. E.g. when investigating blackduck alerts
> (complaining about netty deps for current 3.10.5.Final), it would be cleaner
> to start from a dependency tree where Tez doesn't depend on netty directly in
> order to eliminate its responsibility (and move the focus to underlying
> hadoop for instance).-
> Tez depends on netty3 almost only in ShuffleHandler and some related classes.
> We can eliminate netty3 by upgrading it, but this effort might involve some
> testing due to fundamental [changes from
> netty3->netty4|https://netty.io/wiki/new-and-noteworthy-in-4.0.html] + we
> don't have a reference yet, as [hadoop's
> ShuffleHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java]
> is still on netty3.
> As per the netty documentation, we can also expect some performance
> improvement (e.g. Pooled buffers).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
