[GitHub] incubator-livy issue #114: [LIVY-495] Add thriftserver UI
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/114 cc @vanzin ---
[GitHub] incubator-livy issue #112: [LIVY-511][LIVY-512] Remove support for old Spark...
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/112 @vanzin thanks, I have no more comments apart from https://github.com/apache/incubator-livy/pull/112#discussion_r220647323. I have not reviewed it carefully, but I am still quite new to Livy codebase, so it would take a lot for me to do a throughout review of this PR, which is pretty huge. So once you address that comment I am fine with this PR, thanks. ---
[GitHub] incubator-livy pull request #112: [LIVY-511][LIVY-512] Remove support for ol...
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/incubator-livy/pull/112#discussion_r220122923 --- Diff: README.md --- @@ -57,12 +57,8 @@ Required python packages for building Livy: To run Livy, you will also need a Spark installation. You can get Spark releases at https://spark.apache.org/downloads.html. -Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy -will automatically pick repl dependencies through detecting the Scala version of Spark. - -Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly -switch to different versions of Spark through ``SPARK_HOME`` configuration, without needing to -rebuild Livy. +Livy requires Spark 2.2 or 2.3. You can switch to a different version of Spark by setting the --- End diff -- Is there a specific reason not to state just Spark 2.2+? Do we have problems with 2.4? ---
[GitHub] incubator-livy issue #113: [LIVY-500] Add beeline client for dev
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/113 cc @vanzin ---
[GitHub] incubator-livy pull request #107: [LIVY-494] Add thriftserver to Livy server
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/107#discussion_r219769059
--- Diff: server/src/main/scala/org/apache/livy/server/LivyServer.scala ---
@@ -115,6 +117,16 @@ class LivyServer extends Logging {
error("Failed to run kinit, stopping the server.")
sys.exit(1)
}
+ // This is and should be the only place where a login() on the UGI
is performed.
--- End diff --
Actually this wouldn't really work as - for instance -
`UserGroupInformation.loginUserFromKeytabAndReturnUGI` is ok to be called, as
it doesn't perform a `UserGroupInformation.login()`. We can list all the APIs
which we should not call, but I am not sure it is worth.
---
[GitHub] incubator-livy issue #107: [LIVY-494] Add thriftserver to Livy server
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/107 Travis failure is intermittent, the branch build passed: https://travis-ci.org/mgaido91/incubator-livy/builds/432380763 ---
[GitHub] incubator-livy issue #120: Upgrade jetty version
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/120 @arunmahadevan there is a consistent UT failure. We need to address it. I think we should also have a JIRA (to be referenced here) proposing the Jetty version upgrade. Thanks. ---
[GitHub] incubator-livy pull request #113: [LIVY-500] Add beeline client for dev
Github user mgaido91 closed the pull request at: https://github.com/apache/incubator-livy/pull/113 ---
[GitHub] incubator-livy issue #117: [WIP][LIVY-502] Remove dependency on hive-exec
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/117 thanks for the review @vanzin. I have updated the PR description. Let me know if it still needs to be improved. > I think as a first step getting the Hive code in, with the minimal amount of changes necessary, is better. I am not sure, as the only difference from that was moving from Java to Scala, which saves many conversion back and forth of collections and hence adds unneeded code only to handle this. > overall take another look at whether the things you're forking really need to be forked I am pretty sure that nothing which is there is unneeded. ---
[GitHub] incubator-livy pull request #116: [LIVY-520] Add log when session is GC-ed
Github user mgaido91 closed the pull request at: https://github.com/apache/incubator-livy/pull/116 ---
[GitHub] incubator-livy issue #120: [LIVY-526] Upgrade jetty version
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/120 thanks. LGTM ---
[GitHub] incubator-livy pull request #113: [LIVY-500] Add beeline client for dev
Github user mgaido91 closed the pull request at: https://github.com/apache/incubator-livy/pull/113 ---
[GitHub] incubator-livy pull request #114: [LIVY-495] Add thriftserver UI
GitHub user mgaido91 reopened a pull request: https://github.com/apache/incubator-livy/pull/114 [LIVY-495] Add thriftserver UI ## What changes were proposed in this pull request? The PR adds a new UI tab dedicated to thrift-server sessions. The UI shows the active sessions with a link to the corresponding Livy session. Moreover, the same information is also exposed through a REST endpoint. ## How was this patch tested? Manual tests. A screenshot of the UI is: https://user-images.githubusercontent.com/8821783/46002567-b8ad2e80-c0ae-11e8-8c14-af5af8e67015.png;> You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/incubator-livy LIVY-495 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-livy/pull/114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #114 commit 761bdad5f5734c4bc4a07ffa701fa2ec133e7739 Author: Marco Gaido Date: 2018-09-10T12:18:42Z [LIVY-495] Add thriftserver UI ---
[GitHub] incubator-livy pull request #113: [LIVY-500] Add beeline client for dev
GitHub user mgaido91 reopened a pull request: https://github.com/apache/incubator-livy/pull/113 [LIVY-500] Add beeline client for dev ## What changes were proposed in this pull request? The PR adds a new `thriftserver/client` module and a `beeline` script which are useful to have a client for the thriftserver during local testing. Both things are intended for dev usage only and are not shipped with the distribution. ## How was this patch tested? manual test You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/incubator-livy LIVY-500 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-livy/pull/113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #113 commit a73ebf5dbcd1879678454a184c45bc7d4081a77d Author: Marco Gaido Date: 2018-09-10T11:58:53Z [LIVY-500] Add beeline client for dev ---
[GitHub] incubator-livy pull request #114: [LIVY-495] Add thriftserver UI
Github user mgaido91 closed the pull request at: https://github.com/apache/incubator-livy/pull/114 ---
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/117#discussion_r221881729
--- Diff:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/auth/AuthenticationProvider.scala
---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.auth
+
+import java.lang.reflect.InvocationTargetException
+import javax.security.sasl.AuthenticationException
+
+import org.apache.hive.service.auth.PasswdAuthenticationProvider
+
+import org.apache.livy.LivyConf
+
+object AuthenticationProvider {
--- End diff --
Yes, I agree. Anyway, I don't think there is much difference as of now, ie.
both APIs support Kerberos or none (this supports custom too at the moment, it
i true, but I am not even sure how widespread its adoption is in Hive).
Anyway I think we can revisit this later. This is a very sensitive part and
I think a PR focused only on this would be better.
---
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/117#discussion_r221883081
--- Diff:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/auth/LivyDelegationTokenSecretManager.scala
---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.auth
+
+import java.io.{ByteArrayInputStream, DataInputStream, IOException}
+
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.token.Token
+import
org.apache.hadoop.security.token.delegation.{AbstractDelegationTokenIdentifier,
AbstractDelegationTokenSecretManager}
+
+import org.apache.livy.LivyConf
+
+/**
+ * A secret manager. It is taken from analogous implementation in the
MapReduce client.
+ */
+class LivyDelegationTokenSecretManager(val livyConf: LivyConf)
--- End diff --
I don't think we can remove it. In particular, the `getTokenIdentifier` and
`verifyDelegationToken` are used or the client authentication.
---
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/incubator-livy/pull/117#discussion_r221885113 --- Diff: thriftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftHttpCLIService.scala --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.livy.thriftserver.cli + +import java.util.concurrent.SynchronousQueue +import java.util.concurrent.TimeUnit +import javax.ws.rs.HttpMethod + +import org.apache.hive.service.rpc.thrift.TCLIService +import org.apache.hive.service.server.ThreadFactoryWithGarbageCleanup +import org.apache.thrift.protocol.TBinaryProtocol +import org.eclipse.jetty.server.HttpConfiguration +import org.eclipse.jetty.server.HttpConnectionFactory +import org.eclipse.jetty.server.Server +import org.eclipse.jetty.server.ServerConnector +import org.eclipse.jetty.server.handler.gzip.GzipHandler +import org.eclipse.jetty.servlet.ServletContextHandler +import org.eclipse.jetty.servlet.ServletHolder +import org.eclipse.jetty.util.ssl.SslContextFactory +import org.eclipse.jetty.util.thread.ExecutorThreadPool + +import org.apache.livy.LivyConf +import org.apache.livy.thriftserver.LivyCLIService +import org.apache.livy.thriftserver.auth.AuthFactory + +/** + * This class is ported from Hive. We cannot reuse Hive's one because we need to use the + * `LivyCLIService`, `LivyConf` and `AuthFacotry` instead of Hive's one. + */ +class ThriftHttpCLIService( --- End diff -- I am not sure about this. Here we have a dedicated thread-pool associated with this which is separate from Livy's HTTP server. I am not sure it is feasible. We'd need to try. ---
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/117#discussion_r221887180
--- Diff:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/operation/GetTypeInfoOperation.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.operation
+
+import java.sql.{DatabaseMetaData, Types}
+
+import org.apache.hive.service.cli.{HiveSQLException, OperationState,
OperationType, SessionHandle}
+
+import org.apache.livy.Logging
+import org.apache.livy.thriftserver.serde.ResultSet
+import org.apache.livy.thriftserver.types.{BasicDataType, Field, Schema}
+
+sealed case class TypeInfo(name: String, sqlType: Int, precision:
Option[Int],
+ caseSensitive: Boolean, searchable: Short, unsignedAttribute: Boolean,
numPrecRadix: Option[Int])
+
+/**
+ * GetTypeInfoOperation.
+ *
+ */
+class GetTypeInfoOperation(sessionHandle: SessionHandle)
--- End diff --
not sure bout your question:
- If the question is: what do we need them for? They are used when
answering to metadata calls by the JDBC driver;
- If the question is: can't we reuse Hive's one? No, because Hive's
classes use `HiveSession` and `HiveConf` and we need to get rid of them.
---
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/117#discussion_r221883802
--- Diff:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftCLIService.scala
---
@@ -0,0 +1,745 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.cli
+
+import java.io.IOException
+import java.net.{InetAddress, UnknownHostException}
+import java.util
+import java.util.Collections
+import javax.security.auth.login.LoginException
+
+import scala.collection.JavaConverters._
+
+import com.google.common.base.Preconditions.checkArgument
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.security.authentication.util.KerberosName
+import org.apache.hadoop.security.authorize.ProxyUsers
+import org.apache.hadoop.util.StringUtils
+import org.apache.hive.service.{ServiceException, ServiceUtils}
+import org.apache.hive.service.auth.{HiveAuthConstants,
TSetIpAddressProcessor}
+import org.apache.hive.service.auth.HiveAuthConstants.AuthTypes
+import org.apache.hive.service.cli._
+import org.apache.hive.service.rpc.thrift._
+import org.apache.thrift.TException
+import org.apache.thrift.server.ServerContext
+
+import org.apache.livy.LivyConf
+import org.apache.livy.thriftserver.{LivyCLIService, LivyThriftServer,
SessionInfo, ThriftService}
+import org.apache.livy.thriftserver.auth.AuthFactory
+
+/**
+ * This class is ported from Hive. We cannot reuse Hive's one because we
need to use the
+ * `LivyCLIService`, `LivyConf` and `AuthFacotry` instead of Hive's one.
+ */
+abstract class ThriftCLIService(val cliService: LivyCLIService, val
serviceName: String)
+extends ThriftService(serviceName) with TCLIService.Iface with
Runnable {
+
+ def hiveAuthFactory: AuthFactory
+
+ protected val currentServerContext = new ThreadLocal[ServerContext]
+ protected var portNum: Int = 0
+ protected var serverIPAddress: InetAddress = _
+ protected var hiveHost: String = _
+ private var isStarted: Boolean = false
+ protected var isEmbedded: Boolean = false
+ protected var livyConf: LivyConf = _
+ protected var minWorkerThreads: Int = 0
+ protected var maxWorkerThreads: Int = 0
+ protected var workerKeepAliveTime: Long = 0L
+ private var serverThread: Thread = _
+
+ override def init(conf: LivyConf): Unit = {
+livyConf = conf
+hiveHost = livyConf.get(LivyConf.THRIFT_BIND_HOST)
+try {
+ if (hiveHost == null || hiveHost.isEmpty) {
+serverIPAddress = InetAddress.getLocalHost
+ } else {
+serverIPAddress = InetAddress.getByName(hiveHost)
+ }
+} catch {
+ case e: UnknownHostException =>
+throw new ServiceException(e)
+}
+portNum = livyConf.getInt(LivyConf.THRIFT_SERVER_PORT)
+workerKeepAliveTime =
livyConf.getTimeAsMs(LivyConf.THRIFT_WORKER_KEEPALIVE_TIME) / 1000
+minWorkerThreads = livyConf.getInt(LivyConf.THRIFT_MIN_WORKER_THREADS)
+maxWorkerThreads = livyConf.getInt(LivyConf.THRIFT_MAX_WORKER_THREADS)
+super.init(livyConf)
+ }
+
+ protected def initServer(): Unit
+
+ override def start(): Unit = {
+super.start()
+if (!isStarted && !isEmbedded) {
+ initServer()
+ serverThread = new Thread(this)
+ serverThread.setName("Thrift Server")
+ serverThread.start()
+ isStarted = true
+}
+ }
+
+ protected def stopServer(): Unit
+
+ override def stop(): Unit = {
+if (isStarted && !isEmbedded) {
+ if (serverThread != null) {
+serverThread.interrupt()
+serverThread = null
+ }
+ stopServer()
+ isStarted = false
+}
+super.stop()
+ }
+
+ def getPortNumber: Int = portNum
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/117#discussion_r221886399
--- Diff:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftHttpServlet.scala
---
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.cli
+
+import java.io.IOException
+import java.security.{PrivilegedExceptionAction, SecureRandom}
+import javax.servlet.ServletException
+import javax.servlet.http.{Cookie, HttpServletRequest, HttpServletResponse}
+import javax.ws.rs.core.NewCookie
+
+import scala.collection.JavaConverters._
+
+import org.apache.commons.codec.binary.{Base64, StringUtils}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.security.authentication.util.KerberosName
+import org.apache.hive.service.CookieSigner
+import org.apache.hive.service.auth.{HiveAuthConstants,
HttpAuthenticationException, HttpAuthUtils}
+import org.apache.hive.service.auth.HiveAuthConstants.AuthTypes
+import org.apache.hive.service.cli.HiveSQLException
+import org.apache.thrift.TProcessor
+import org.apache.thrift.protocol.TProtocolFactory
+import org.apache.thrift.server.TServlet
+import org.ietf.jgss.{GSSContext, GSSCredential, GSSException, GSSManager,
Oid}
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.thriftserver.SessionInfo
+import org.apache.livy.thriftserver.auth.{AuthenticationProvider,
AuthFactory}
+
+class ThriftHttpServlet(
+processor: TProcessor,
+protocolFactory: TProtocolFactory,
+val authType: String,
+val serviceUGI: UserGroupInformation,
+val httpUGI: UserGroupInformation,
+val authFactory: AuthFactory,
+val livyConf: LivyConf) extends TServlet(processor, protocolFactory)
with Logging {
+
+ private val isCookieAuthEnabled =
livyConf.getBoolean(LivyConf.THRIFT_HTTP_COOKIE_AUTH_ENABLED)
+
+ // Class members for cookie based authentication.
+ private val signer: CookieSigner = if (isCookieAuthEnabled) {
+ // Generate the signer with secret.
+ val secret = ThriftHttpServlet.RAN.nextLong.toString
+ debug("Using the random number as the secret for cookie generation "
+ secret)
+ new CookieSigner(secret.getBytes())
+} else {
+ null
+}
+
+ private val cookieDomain =
livyConf.get(LivyConf.THRIFT_HTTP_COOKIE_DOMAIN)
+ private val cookiePath = livyConf.get(LivyConf.THRIFT_HTTP_COOKIE_PATH)
+ private val cookieMaxAge =
+(livyConf.getTimeAsMs(LivyConf.THRIFT_HTTP_COOKIE_MAX_AGE) /
1000).toInt
+ private val isCookieSecure = livyConf.getBoolean(LivyConf.THRIFT_USE_SSL)
+ private val isHttpOnlyCookie =
livyConf.getBoolean(LivyConf.THRIFT_HTTP_COOKIE_IS_HTTPONLY)
+ private val xsrfFilterEnabled =
livyConf.getBoolean(LivyConf.THRIFT_XSRF_FILTER_ENABLED)
+
+ @throws[IOException]
+ @throws[ServletException]
+ override protected def doPost(
+ request: HttpServletRequest, response: HttpServletResponse): Unit = {
+var clientUserName: String = null
+var requireNewCookie: Boolean = false
+
+try {
+ if (xsrfFilterEnabled) {
+val continueProcessing = ThriftHttpServlet.doXsrfFilter(request,
response)
+if (!continueProcessing) {
+ warn("Request did not have valid XSRF header, rejecting.")
+ return
+}
+ }
+ // If the cookie based authentication is already enabled, parse the
+ // request and validate the request cookies.
+ if (isCookieAuthEnabled) {
+clientUserName = validateCookie(request)
+requireNewCookie = clientUserName == null
+if (requireNewCookie) {
+ info("Could not validate cookie sent, will try to generate a new
cookie")
+}
+ }
+ //
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/117#discussion_r221890669
--- Diff:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/auth/AuthBridgeServer.scala
---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.auth
+
+import java.io.IOException
+import java.net.InetAddress
+import java.security.{PrivilegedAction, PrivilegedExceptionAction}
+import java.util
+import javax.security.auth.callback.{Callback, CallbackHandler,
NameCallback, PasswordCallback, UnsupportedCallbackException}
+import javax.security.sasl.{AuthorizeCallback, RealmCallback, SaslServer}
+
+import org.apache.commons.codec.binary.Base64
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hadoop.security.{SaslRpcServer, UserGroupInformation}
+import org.apache.hadoop.security.SaslRpcServer.AuthMethod
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
+import org.apache.hadoop.security.token.SecretManager.InvalidToken
+import org.apache.thrift.{TException, TProcessor}
+import org.apache.thrift.protocol.TProtocol
+import org.apache.thrift.transport.{TSaslServerTransport, TSocket,
TTransport, TTransportException, TTransportFactory}
+
+import org.apache.livy.Logging
+
+/**
+ * The class is taken from Hive's `HadoopThriftAuthBridge.Server`. It
bridges Thrift's SASL
+ * transports to Hadoop's SASL callback handlers and authentication
classes.
+ *
+ * This class is based on Hive's one.
+ */
+class AuthBridgeServer(private val secretManager:
LivyDelegationTokenSecretManager) {
+ private val ugi = try {
+ UserGroupInformation.getCurrentUser
+} catch {
+ case ioe: IOException => throw new TTransportException(ioe)
+}
+
+ /**
+ * Create a TTransportFactory that, upon connection of a client socket,
+ * negotiates a Kerberized SASL transport. The resulting
TTransportFactory
+ * can be passed as both the input and output transport factory when
+ * instantiating a TThreadPoolServer, for example.
+ *
+ * @param saslProps Map of SASL properties
+ */
+ @throws[TTransportException]
+ def createTransportFactory(saslProps: util.Map[String, String]):
TTransportFactory = {
+val transFactory: TSaslServerTransport.Factory =
createSaslServerTransportFactory(saslProps)
+new TUGIAssumingTransportFactory(transFactory, ugi)
+ }
+
+ /**
+ * Create a TSaslServerTransport.Factory that, upon connection of a
client
+ * socket, negotiates a Kerberized SASL transport.
+ *
+ * @param saslProps Map of SASL properties
+ */
+ @throws[TTransportException]
+ def createSaslServerTransportFactory(
+ saslProps: util.Map[String, String]): TSaslServerTransport.Factory =
{
+// Parse out the kerberos principal, host, realm.
+val kerberosName: String = ugi.getUserName
+val names: Array[String] =
SaslRpcServer.splitKerberosName(kerberosName)
+if (names.length != 3) {
+ throw new TTransportException(s"Kerberos principal should have 3
parts: $kerberosName")
+}
+val transFactory: TSaslServerTransport.Factory = new
TSaslServerTransport.Factory
+transFactory.addServerDefinition(AuthMethod.KERBEROS.getMechanismName,
+ names(0), names(1), // two parts of kerberos principal
+ saslProps,
+ new SaslRpcServer.SaslGssCallbackHandler)
+transFactory.addServerDefinition(AuthMethod.TOKEN.getMechanismName,
+ null,
+ SaslRpcServer.SASL_DEFAULT_REALM,
+ saslProps,
+ new SaslDigestCallbackHandler(secretManager))
+transFactory
+ }
+
+ /**
+ * Wrap a TTransportFactory in such a way that, before processing any
RPC, it
+ * assumes the UserGroupInformation of the user authenticated by
+ * the SASL
[GitHub] incubator-livy pull request #116: [LIVY-520] Add log when session is GC-ed
GitHub user mgaido91 reopened a pull request: https://github.com/apache/incubator-livy/pull/116 [LIVY-520] Add log when session is GC-ed ## What changes were proposed in this pull request? When a session is inactive for a long time, it is GC-ed, ie. it is closed. Currently this operation is not logged, so it is not easy to understand why a session was closed. The PR proposes to add a log in order to explain what happened. ## How was this patch tested? manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/incubator-livy LIVY-520 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-livy/pull/116.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #116 commit 8a11292cbed0c96bb4b3f7e5906a03d3d479d2b0 Author: Marco Gaido Date: 2018-09-27T10:14:52Z [LIVY-520] Add log when session is GC-ed ---
[GitHub] incubator-livy issue #116: [LIVY-520] Add log when session is GC-ed
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/116 cc @alex-the-man @jerryshao @vanzin ---
[GitHub] incubator-livy issue #117: [WIP][LIVY-502] Remove dependency on hive-exec
Github user mgaido91 commented on the issue: https://github.com/apache/incubator-livy/pull/117 cc @jerryshao @vanzin ---
[GitHub] incubator-livy pull request #117: [WIP][LIVY-502] Remove dependency on hive-...
GitHub user mgaido91 opened a pull request: https://github.com/apache/incubator-livy/pull/117 [WIP][LIVY-502] Remove dependency on hive-exec ## What changes were proposed in this pull request? This PR removes the dependency on `hive-exec`. Only modules of Hive which are used after this PR are `hive-service-rpc` and `hive-service`. This drastically reduces the amount of JARs needed by the thriftserver module. ## How was this patch tested? existing UTs + manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/incubator-livy LIVY-502 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-livy/pull/117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #117 commit e7aa5b1c98e322fb60963bcca18965c5d874ce10 Author: Marco Gaido Date: 2018-09-28T12:07:38Z [WIP][LIVY-502] Remove dependency on hive-exec ---
