[
https://issues.apache.org/jira/browse/LIVY-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyorgy Gal updated LIVY-737:
----------------------------
Fix Version/s: 0.10.0
(was: 0.9.0)
This issue has been moved to the 0.10.0 release as part of a bulk update. If
you feel this is moved out inappropriately, feel free to provide justification
and reset the Fix Version to 0.9.0.
> Livy Server support use hadoop native method in local filesystem
> ----------------------------------------------------------------
>
> Key: LIVY-737
> URL: https://issues.apache.org/jira/browse/LIVY-737
> Project: Livy
> Issue Type: New Feature
> Components: Server
> Reporter: Zhefeng Wang
> Assignee: Zhefeng Wang
> Priority: Minor
> Fix For: 0.10.0
>
>
> Livy server doesn't use hadoop local methods when choosing local file system
> to store sessionMetadata and nextId，which spent more time than using hadoop
> local methods.
> this will affect livy server throughput in high-concurrency scenario, because
> sessionStore.nextId is synchronized and will retain the SessionManager Object
> lock.
>
> {code:java}
> //代码占位符
> "qtp20084184-405665" #405665 prio=5 os_prio=0 tid=0x00007fa9dca37000
> nid=0xf55d runnable [0x00007faa15900000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
> at org.apache.hadoop.util.Shell.run(Shell.java:456)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
> at org.apache.hadoop.fs.FileUtil.readLink(FileUtil.java:160)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileLinkStatusInternal(RawLocalFileSystem.java:835)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatus(RawLocalFileSystem.java:795)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.getFileLinkStatus(DelegateToFileSystem.java:130)
> at
> org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:705)
> at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)
> at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)
> at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)at
> org.apache.livy.server.recovery.FileSystemStateStore$$anonfun$set$1.apply(FileSystemStateStore.scala:92)
> at
> org.apache.livy.server.recovery.FileSystemStateStore$$anonfun$set$1.apply(FileSystemStateStore.scala:88)
> at org.apache.livy.Utils$.usingResource(Utils.scala:103)
> at
> org.apache.livy.server.recovery.FileSystemStateStore.set(FileSystemStateStore.scala:88)
> at
> org.apache.livy.server.recovery.SessionStore.saveNextSessionId(SessionStore.scala:50)
> at org.apache.livy.sessions.SessionManager.nextId(SessionManager.scala:93)
> - locked <0x00000000804f1228> (a org.apache.livy.sessions.BatchSessionManager)
> at
> org.apache.livy.server.batch.BatchSessionServlet.createSession(BatchSessionServlet.scala:68)
> at
> org.apache.livy.server.batch.BatchSessionServlet.createSession(BatchSessionServlet.scala:40)
> at
> org.apache.livy.server.SessionServlet$$anonfun$17.apply(SessionServlet.scala:161)
> at
> org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
