[
https://issues.apache.org/jira/browse/HIVE-24524?focusedWorklogId=581674&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-581674
]
ASF GitHub Bot logged work on HIVE-24524:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Apr/21 10:27
Start Date: 13/Apr/21 10:27
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #1778:
URL: https://github.com/apache/hive/pull/1778#discussion_r612323669
##########
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
##########
@@ -339,27 +350,60 @@ private ShuffleHandler(Configuration conf) {
public void start() throws Exception {
- ServerBootstrap bootstrap = new ServerBootstrap(selector);
- // Timer is shared across entire factory and must be released separately
- timer = new HashedWheelTimer();
- try {
- pipelineFact = new HttpPipelineFactory(conf, timer);
- } catch (Exception ex) {
- throw new RuntimeException(ex);
- }
- bootstrap.setPipelineFactory(pipelineFact);
- bootstrap.setOption("backlog", NetUtil.SOMAXCONN);
+ ServerBootstrap bootstrap = new ServerBootstrap()
+ .channel(NioServerSocketChannel.class)
+ .group(bossGroup, workerGroup)
+ .localAddress(port)
+ .option(ChannelOption.SO_BACKLOG, NetUtil.SOMAXCONN)
+ .childOption(ChannelOption.SO_KEEPALIVE, true);
+ initPipeline(bootstrap, conf);
+
port = conf.getInt(SHUFFLE_PORT_CONFIG_KEY, DEFAULT_SHUFFLE_PORT);
- Channel ch = bootstrap.bind(new InetSocketAddress(port));
+ Channel ch = bootstrap.bind().sync().channel();
accepted.add(ch);
- port = ((InetSocketAddress)ch.getLocalAddress()).getPort();
+ port = ((InetSocketAddress)ch.localAddress()).getPort();
conf.set(SHUFFLE_PORT_CONFIG_KEY, Integer.toString(port));
- pipelineFact.SHUFFLE.setPort(port);
+ SHUFFLE.setPort(port);
if (dirWatcher != null) {
dirWatcher.start();
}
- LOG.info("LlapShuffleHandler" + " listening on port " + port + "
(SOMAXCONN: " + bootstrap.getOption("backlog")
- + ")");
+ LOG.info("LlapShuffleHandler listening on port {} (SOMAXCONN: {})", port,
NetUtil.SOMAXCONN);
+ }
+
+ private void initPipeline(ServerBootstrap bootstrap, Configuration conf)
throws Exception {
+ SHUFFLE = getShuffle(conf);
+ // TODO Setup SSL Shuffle
Review comment:
I think we don't support SSL shuffle for LLAP at the moment (+ the
comment is quite old), e.g. Cloudera's data warehouse, ssl on shuffle is
handled transparently by the environment
I haven't touched this part in this patch, and not even sure what's the plan
:) that's why I simply kept this as is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 581674)
Time Spent: 1h (was: 50m)
> LLAP ShuffleHandler: upgrade to netty4
> --------------------------------------
>
> Key: HIVE-24524
> URL: https://issues.apache.org/jira/browse/HIVE-24524
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Tez already has a WIP patch for upgrading its shuffle handler to netty4.
> Netty4 is told to be a possible performance improvement compared to Netty3.
> However, the refactor is not trivial, TEZ-4157 covers that more or less (the
> code bases are very similar).
> Background:
> netty4 migration guideline:
> https://netty.io/wiki/new-and-noteworthy-in-4.0.html
> articles of possible performance improvement:
> https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html
> https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/
> some other notes: Netty3 is EOL since 2016:
> https://netty.io/news/2016/06/29/3-10-6-Final.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
