Re: Missing board report

[Marcelo Vanzin]

Hi Justin,

Sorry, it wasn't intentional, it was probably just not in the original
report I based my update on.

I guess we just haven't been asking anything from the mentors, so we
should just mention that in the report?

On Tue, Oct 2, 2018 at 3:40 PM Justin Mclean  wrote:
>
> HI,
>
> I notice you didn't answer this question:
>
> Have your mentors been helpful and responsive or are things falling
> through the cracks? In the latter case, please list any open issues
> that need to be addressed.
>
> And removed it from your report. Any reason why?
>
> Thanks,
> Justin



-- 
Marcelo

Re: Missing board report

[Marcelo Vanzin]

I'm pretty sure I don't have access to the wiki; but I took the last report
and added a small blurb about what's been going on. Feel free to update the
wiki on my behalf.

===

Livy is web service that exposes a REST interface for managing long running
Apache Spark contexts in your cluster. With Livy, new applications can be
built on top of Apache Spark that require fine grained interaction with many
Spark contexts.

Livy has been incubating since 2017-06-05.

Three most important issues to address in the move towards graduation:

  1. Grow the community to have more activities.
  2. Grow more contributors and committers.
  3. Regular release.


Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

None


How has the community developed since the last report?

Activity on the dev and user mailing lists as well as the rate of reported
JIRA issues has stayed steady. There has been an uptick of development
activity around adding a new feature to the project (a Hive-compatible
Thrift
server implementation).


How has the project developed since the last report?

A new feature (Hive-compatible Thrift server) is being implemented, and some
code to support legacy releases of Java, Scala and Spark has been removed.


How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [ ] Working towards first release
  [X] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  2018-02-06

When were the last committers or PPMC members elected?

  2017-09-18

Signed-off-by:

  [ ](livy) Bikas Saha
 Comments:
  [ ](livy) Brock Noland
 Comments:
  [ ](livy) Luciano Resende
 Comments:
  [ ](livy) Jean-Baptiste Onofre
 Comments:


On Tue, Oct 2, 2018 at 2:13 PM Alex Bozarth  wrote:

> @marcelo and @jerry you've been reviewing and merging the most recent code
> changes as I've been a bit too busy. I know we've added a thrift server in
> the last quarter, could one of you fill out the report this time? If I
> don't hear back by end of work day then I'll just fill it out to the best
> of my understanding.
>
>
> *Alex Bozarth*
> Software Engineer
> Center for Open-Source Data & AI Technologies
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Justin Mclean ---10/02/2018 01:54:50
> PM---HI, Just a reminder that the report is due today and I don']Justin
> Mclean ---10/02/2018 01:54:50 PM---HI, Just a reminder that the report is
> due today and I don't see any discussion on it. It helps me f
>
> From: Justin Mclean 
> To: 
> Date: 10/02/2018 01:54 PM
> Subject: Re: Missing board report
> --
>
>
>
> HI,
>
> Just a reminder that the report is due today and I don't see any
> discussion on it. It helps me finish the IPMC report on time if you get
> your report in on time.
>
> Thanks,
> Justin
>
>
>
>
>

-- 
Marcelo

Re: Missing board report

[Alex Bozarth]

@marcelo and @jerry you've been reviewing and merging the most recent code
changes as I've been a bit too busy. I know we've added a thrift server in
the last quarter, could one of you fill out the report this time? If I
don't hear back by end of work day then I'll just fill it out to the best
of my understanding.


   
 Alex Bozarth   
   
 Software Engineer  
   
 Center for Open-Source Data & AI Technologies  
   

   

 

 

 
 E-mail: ajboz...@us.ibm.com
 
 GitHub: github.com/ajbozarth   
 
   505 Howard 
Street 
 San Francisco, CA 
94105 
   United 
States 

 








From:   Justin Mclean 
To: 
Date:   10/02/2018 01:54 PM
Subject:Re: Missing board report



HI,

Just a reminder that the report is due today and I don't see any discussion
on it. It helps me finish the IPMC report on time if you get your report in
on time.

Thanks,
Justin

Re: Missing board report

[Luciano Resende]

Livy team, the report is not optional... Could someone please help draft a
report... it should take a few minutes and we can help posting if you don’t
have access.

Thanks

On Tue, Oct 2, 2018 at 16:54 Justin Mclean  wrote:

> HI,
>
> Just a reminder that the report is due today and I don't see any
> discussion on it. It helps me finish the IPMC report on time if you get
> your report in on time.
>
> Thanks,
> Justin
>
-- 
Sent from my Mobile device

Re: Missing board report

[Justin Mclean]

HI,

Just a reminder that the report is due today and I don't see any discussion on 
it. It helps me finish the IPMC report on time if you get your report in on 
time.

Thanks,
Justin

[GitHub] incubator-livy pull request #113: [LIVY-500] Add beeline client for dev

[mgaido91]

Github user mgaido91 closed the pull request at:

https://github.com/apache/incubator-livy/pull/113


---

[GitHub] incubator-livy issue #117: [WIP][LIVY-502] Remove dependency on hive-exec

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/incubator-livy/pull/117
  
thanks for the review @vanzin. I have updated the PR description. Let me 
know if it still needs to be improved.

>  I think as a first step getting the Hive code in, with the minimal 
amount of changes necessary, is better.

I am not sure, as the only difference from that was moving from Java to 
Scala, which saves many conversion back and forth of collections and hence adds 
unneeded code only to handle this.

> overall take another look at whether the things you're forking really 
need to be forked

I am pretty sure that nothing which is there is unneeded.



---

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221890669
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/auth/AuthBridgeServer.scala
 ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.auth
+
+import java.io.IOException
+import java.net.InetAddress
+import java.security.{PrivilegedAction, PrivilegedExceptionAction}
+import java.util
+import javax.security.auth.callback.{Callback, CallbackHandler, 
NameCallback, PasswordCallback, UnsupportedCallbackException}
+import javax.security.sasl.{AuthorizeCallback, RealmCallback, SaslServer}
+
+import org.apache.commons.codec.binary.Base64
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hadoop.security.{SaslRpcServer, UserGroupInformation}
+import org.apache.hadoop.security.SaslRpcServer.AuthMethod
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+import org.apache.hadoop.security.token.SecretManager.InvalidToken
+import org.apache.thrift.{TException, TProcessor}
+import org.apache.thrift.protocol.TProtocol
+import org.apache.thrift.transport.{TSaslServerTransport, TSocket, 
TTransport, TTransportException, TTransportFactory}
+
+import org.apache.livy.Logging
+
+/**
+ * The class is taken from Hive's `HadoopThriftAuthBridge.Server`. It 
bridges Thrift's SASL
+ * transports to Hadoop's SASL callback handlers and authentication 
classes.
+ *
+ * This class is based on Hive's one.
+ */
+class AuthBridgeServer(private val secretManager: 
LivyDelegationTokenSecretManager) {
+  private val ugi = try {
+  UserGroupInformation.getCurrentUser
+} catch {
+  case ioe: IOException => throw new TTransportException(ioe)
+}
+
+  /**
+   * Create a TTransportFactory that, upon connection of a client socket,
+   * negotiates a Kerberized SASL transport. The resulting 
TTransportFactory
+   * can be passed as both the input and output transport factory when
+   * instantiating a TThreadPoolServer, for example.
+   *
+   * @param saslProps Map of SASL properties
+   */
+  @throws[TTransportException]
+  def createTransportFactory(saslProps: util.Map[String, String]): 
TTransportFactory = {
+val transFactory: TSaslServerTransport.Factory = 
createSaslServerTransportFactory(saslProps)
+new TUGIAssumingTransportFactory(transFactory, ugi)
+  }
+
+  /**
+   * Create a TSaslServerTransport.Factory that, upon connection of a 
client
+   * socket, negotiates a Kerberized SASL transport.
+   *
+   * @param saslProps Map of SASL properties
+   */
+  @throws[TTransportException]
+  def createSaslServerTransportFactory(
+  saslProps: util.Map[String, String]): TSaslServerTransport.Factory = 
{
+// Parse out the kerberos principal, host, realm.
+val kerberosName: String = ugi.getUserName
+val names: Array[String] = 
SaslRpcServer.splitKerberosName(kerberosName)
+if (names.length != 3) {
+  throw new TTransportException(s"Kerberos principal should have 3 
parts: $kerberosName")
+}
+val transFactory: TSaslServerTransport.Factory = new 
TSaslServerTransport.Factory
+transFactory.addServerDefinition(AuthMethod.KERBEROS.getMechanismName,
+  names(0), names(1), // two parts of kerberos principal
+  saslProps,
+  new SaslRpcServer.SaslGssCallbackHandler)
+transFactory.addServerDefinition(AuthMethod.TOKEN.getMechanismName,
+  null,
+  SaslRpcServer.SASL_DEFAULT_REALM,
+  saslProps,
+  new SaslDigestCallbackHandler(secretManager))
+transFactory
+  }
+
+  /**
+   * Wrap a TTransportFactory in such a way that, before processing any 
RPC, it
+   * assumes the UserGroupInformation of the user authenticated by
+   * the SASL transport.
+   */
+  def 

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221887180
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/operation/GetTypeInfoOperation.scala
 ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.operation
+
+import java.sql.{DatabaseMetaData, Types}
+
+import org.apache.hive.service.cli.{HiveSQLException, OperationState, 
OperationType, SessionHandle}
+
+import org.apache.livy.Logging
+import org.apache.livy.thriftserver.serde.ResultSet
+import org.apache.livy.thriftserver.types.{BasicDataType, Field, Schema}
+
+sealed case class TypeInfo(name: String, sqlType: Int, precision: 
Option[Int],
+  caseSensitive: Boolean, searchable: Short, unsignedAttribute: Boolean, 
numPrecRadix: Option[Int])
+
+/**
+ * GetTypeInfoOperation.
+ *
+ */
+class GetTypeInfoOperation(sessionHandle: SessionHandle)
--- End diff --

not sure bout your question:
 - If the question is: what do we need them for? They are used when 
answering to metadata calls by the JDBC driver;
 - If the question is: can't we reuse Hive's one? No, because Hive's 
classes use `HiveSession` and `HiveConf` and we need to get rid of them.


---

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221886399
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftHttpServlet.scala
 ---
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.cli
+
+import java.io.IOException
+import java.security.{PrivilegedExceptionAction, SecureRandom}
+import javax.servlet.ServletException
+import javax.servlet.http.{Cookie, HttpServletRequest, HttpServletResponse}
+import javax.ws.rs.core.NewCookie
+
+import scala.collection.JavaConverters._
+
+import org.apache.commons.codec.binary.{Base64, StringUtils}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.security.authentication.util.KerberosName
+import org.apache.hive.service.CookieSigner
+import org.apache.hive.service.auth.{HiveAuthConstants, 
HttpAuthenticationException, HttpAuthUtils}
+import org.apache.hive.service.auth.HiveAuthConstants.AuthTypes
+import org.apache.hive.service.cli.HiveSQLException
+import org.apache.thrift.TProcessor
+import org.apache.thrift.protocol.TProtocolFactory
+import org.apache.thrift.server.TServlet
+import org.ietf.jgss.{GSSContext, GSSCredential, GSSException, GSSManager, 
Oid}
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.thriftserver.SessionInfo
+import org.apache.livy.thriftserver.auth.{AuthenticationProvider, 
AuthFactory}
+
+class ThriftHttpServlet(
+processor: TProcessor,
+protocolFactory: TProtocolFactory,
+val authType: String,
+val serviceUGI: UserGroupInformation,
+val httpUGI: UserGroupInformation,
+val authFactory: AuthFactory,
+val livyConf: LivyConf) extends TServlet(processor, protocolFactory) 
with Logging {
+
+  private val isCookieAuthEnabled = 
livyConf.getBoolean(LivyConf.THRIFT_HTTP_COOKIE_AUTH_ENABLED)
+
+  // Class members for cookie based authentication.
+  private val signer: CookieSigner = if (isCookieAuthEnabled) {
+  // Generate the signer with secret.
+  val secret = ThriftHttpServlet.RAN.nextLong.toString
+  debug("Using the random number as the secret for cookie generation " 
+ secret)
+  new CookieSigner(secret.getBytes())
+} else {
+  null
+}
+
+  private val cookieDomain = 
livyConf.get(LivyConf.THRIFT_HTTP_COOKIE_DOMAIN)
+  private val cookiePath = livyConf.get(LivyConf.THRIFT_HTTP_COOKIE_PATH)
+  private val cookieMaxAge =
+(livyConf.getTimeAsMs(LivyConf.THRIFT_HTTP_COOKIE_MAX_AGE) / 
1000).toInt
+  private val isCookieSecure = livyConf.getBoolean(LivyConf.THRIFT_USE_SSL)
+  private val isHttpOnlyCookie = 
livyConf.getBoolean(LivyConf.THRIFT_HTTP_COOKIE_IS_HTTPONLY)
+  private val xsrfFilterEnabled = 
livyConf.getBoolean(LivyConf.THRIFT_XSRF_FILTER_ENABLED)
+
+  @throws[IOException]
+  @throws[ServletException]
+  override protected def doPost(
+  request: HttpServletRequest, response: HttpServletResponse): Unit = {
+var clientUserName: String = null
+var requireNewCookie: Boolean = false
+
+try {
+  if (xsrfFilterEnabled) {
+val continueProcessing = ThriftHttpServlet.doXsrfFilter(request, 
response)
+if (!continueProcessing) {
+  warn("Request did not have valid XSRF header, rejecting.")
+  return
+}
+  }
+  // If the cookie based authentication is already enabled, parse the
+  // request and validate the request cookies.
+  if (isCookieAuthEnabled) {
+clientUserName = validateCookie(request)
+requireNewCookie = clientUserName == null
+if (requireNewCookie) {
+  info("Could not validate cookie sent, will try to generate a new 
cookie")
+}
+  }
+  // If the cookie based authentication is not enabled or the 

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221885113
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftHttpCLIService.scala
 ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.cli
+
+import java.util.concurrent.SynchronousQueue
+import java.util.concurrent.TimeUnit
+import javax.ws.rs.HttpMethod
+
+import org.apache.hive.service.rpc.thrift.TCLIService
+import org.apache.hive.service.server.ThreadFactoryWithGarbageCleanup
+import org.apache.thrift.protocol.TBinaryProtocol
+import org.eclipse.jetty.server.HttpConfiguration
+import org.eclipse.jetty.server.HttpConnectionFactory
+import org.eclipse.jetty.server.Server
+import org.eclipse.jetty.server.ServerConnector
+import org.eclipse.jetty.server.handler.gzip.GzipHandler
+import org.eclipse.jetty.servlet.ServletContextHandler
+import org.eclipse.jetty.servlet.ServletHolder
+import org.eclipse.jetty.util.ssl.SslContextFactory
+import org.eclipse.jetty.util.thread.ExecutorThreadPool
+
+import org.apache.livy.LivyConf
+import org.apache.livy.thriftserver.LivyCLIService
+import org.apache.livy.thriftserver.auth.AuthFactory
+
+/**
+ * This class is ported from Hive. We cannot reuse Hive's one because we 
need to use the
+ * `LivyCLIService`, `LivyConf` and `AuthFacotry` instead of Hive's one.
+ */
+class ThriftHttpCLIService(
--- End diff --

I am not sure about this. Here we have a dedicated thread-pool associated 
with this which is separate from Livy's HTTP server. I am not sure it is 
feasible. We'd need to try.


---

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221883802
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftCLIService.scala
 ---
@@ -0,0 +1,745 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.cli
+
+import java.io.IOException
+import java.net.{InetAddress, UnknownHostException}
+import java.util
+import java.util.Collections
+import javax.security.auth.login.LoginException
+
+import scala.collection.JavaConverters._
+
+import com.google.common.base.Preconditions.checkArgument
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.security.authentication.util.KerberosName
+import org.apache.hadoop.security.authorize.ProxyUsers
+import org.apache.hadoop.util.StringUtils
+import org.apache.hive.service.{ServiceException, ServiceUtils}
+import org.apache.hive.service.auth.{HiveAuthConstants, 
TSetIpAddressProcessor}
+import org.apache.hive.service.auth.HiveAuthConstants.AuthTypes
+import org.apache.hive.service.cli._
+import org.apache.hive.service.rpc.thrift._
+import org.apache.thrift.TException
+import org.apache.thrift.server.ServerContext
+
+import org.apache.livy.LivyConf
+import org.apache.livy.thriftserver.{LivyCLIService, LivyThriftServer, 
SessionInfo, ThriftService}
+import org.apache.livy.thriftserver.auth.AuthFactory
+
+/**
+ * This class is ported from Hive. We cannot reuse Hive's one because we 
need to use the
+ * `LivyCLIService`, `LivyConf` and `AuthFacotry` instead of Hive's one.
+ */
+abstract class ThriftCLIService(val cliService: LivyCLIService, val 
serviceName: String)
+extends ThriftService(serviceName) with TCLIService.Iface with 
Runnable {
+
+  def hiveAuthFactory: AuthFactory
+
+  protected val currentServerContext = new ThreadLocal[ServerContext]
+  protected var portNum: Int = 0
+  protected var serverIPAddress: InetAddress = _
+  protected var hiveHost: String = _
+  private var isStarted: Boolean = false
+  protected var isEmbedded: Boolean = false
+  protected var livyConf: LivyConf = _
+  protected var minWorkerThreads: Int = 0
+  protected var maxWorkerThreads: Int = 0
+  protected var workerKeepAliveTime: Long = 0L
+  private var serverThread: Thread = _
+
+  override def init(conf: LivyConf): Unit = {
+livyConf = conf
+hiveHost = livyConf.get(LivyConf.THRIFT_BIND_HOST)
+try {
+  if (hiveHost == null || hiveHost.isEmpty) {
+serverIPAddress = InetAddress.getLocalHost
+  } else {
+serverIPAddress = InetAddress.getByName(hiveHost)
+  }
+} catch {
+  case e: UnknownHostException =>
+throw new ServiceException(e)
+}
+portNum = livyConf.getInt(LivyConf.THRIFT_SERVER_PORT)
+workerKeepAliveTime = 
livyConf.getTimeAsMs(LivyConf.THRIFT_WORKER_KEEPALIVE_TIME) / 1000
+minWorkerThreads = livyConf.getInt(LivyConf.THRIFT_MIN_WORKER_THREADS)
+maxWorkerThreads = livyConf.getInt(LivyConf.THRIFT_MAX_WORKER_THREADS)
+super.init(livyConf)
+  }
+
+  protected def initServer(): Unit
+
+  override def start(): Unit = {
+super.start()
+if (!isStarted && !isEmbedded) {
+  initServer()
+  serverThread = new Thread(this)
+  serverThread.setName("Thrift Server")
+  serverThread.start()
+  isStarted = true
+}
+  }
+
+  protected def stopServer(): Unit
+
+  override def stop(): Unit = {
+if (isStarted && !isEmbedded) {
+  if (serverThread != null) {
+serverThread.interrupt()
+serverThread = null
+  }
+  stopServer()
+  isStarted = false
+}
+super.stop()
+  }
+
+  def getPortNumber: Int = portNum
+
+  def getServerIPAddress: InetAddress = 

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221883081
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/auth/LivyDelegationTokenSecretManager.scala
 ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.auth
+
+import java.io.{ByteArrayInputStream, DataInputStream, IOException}
+
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.token.Token
+import 
org.apache.hadoop.security.token.delegation.{AbstractDelegationTokenIdentifier, 
AbstractDelegationTokenSecretManager}
+
+import org.apache.livy.LivyConf
+
+/**
+ * A secret manager. It is taken from analogous implementation in the 
MapReduce client.
+ */
+class LivyDelegationTokenSecretManager(val livyConf: LivyConf)
--- End diff --

I don't think we can remove it. In particular, the `getTokenIdentifier` and 
`verifyDelegationToken` are used or the client authentication.


---

[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/117#discussion_r221881729
  
--- Diff: 
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/auth/AuthenticationProvider.scala
 ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.auth
+
+import java.lang.reflect.InvocationTargetException
+import javax.security.sasl.AuthenticationException
+
+import org.apache.hive.service.auth.PasswdAuthenticationProvider
+
+import org.apache.livy.LivyConf
+
+object AuthenticationProvider {
--- End diff --

Yes, I agree. Anyway, I don't think there is much difference as of now, ie. 
both APIs support Kerberos or none (this supports custom too at the moment, it 
i true, but I am not even sure how widespread its adoption is in Hive).

Anyway I think we can revisit this later. This is a very sensitive part and 
I think a PR focused only on this would be better.


---