Zhefeng Wang created LIVY-737:
---------------------------------
Summary: Livy Server support use hadoop native method in local
filesystem
Key: LIVY-737
URL: https://issues.apache.org/jira/browse/LIVY-737
Project: Livy
Issue Type: New Feature
Components: Server
Reporter: Zhefeng Wang
Assignee: Zhefeng Wang
Livy server doesn't use hadoop local methods when choosing local file system to
store sessionMetadata and nextId，which spent more time than using hadoop local
methods.
this will affect livy server throughput in high-concurrency scenario, because
sessionStore.nextId is synchronized and will retain the SessionManager Object
lock.
{code:java}
//代码占位符
"qtp20084184-405665" #405665 prio=5 os_prio=0 tid=0x00007fa9dca37000 nid=0xf55d
runnable [0x00007faa15900000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
at org.apache.hadoop.fs.FileUtil.readLink(FileUtil.java:160)
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileLinkStatusInternal(RawLocalFileSystem.java:835)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatus(RawLocalFileSystem.java:795)
at
org.apache.hadoop.fs.DelegateToFileSystem.getFileLinkStatus(DelegateToFileSystem.java:130)
at
org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:705)
at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)
at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)
at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)at
org.apache.livy.server.recovery.FileSystemStateStore$$anonfun$set$1.apply(FileSystemStateStore.scala:92)
at
org.apache.livy.server.recovery.FileSystemStateStore$$anonfun$set$1.apply(FileSystemStateStore.scala:88)
at org.apache.livy.Utils$.usingResource(Utils.scala:103)
at
org.apache.livy.server.recovery.FileSystemStateStore.set(FileSystemStateStore.scala:88)
at
org.apache.livy.server.recovery.SessionStore.saveNextSessionId(SessionStore.scala:50)
at org.apache.livy.sessions.SessionManager.nextId(SessionManager.scala:93)
- locked <0x00000000804f1228> (a org.apache.livy.sessions.BatchSessionManager)
at
org.apache.livy.server.batch.BatchSessionServlet.createSession(BatchSessionServlet.scala:68)
at
org.apache.livy.server.batch.BatchSessionServlet.createSession(BatchSessionServlet.scala:40)
at
org.apache.livy.server.SessionServlet$$anonfun$17.apply(SessionServlet.scala:161)
at
org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
