[jira] [Resolved] (LIVY-754) precision and scale are not encoded in decimal type
[
https://issues.apache.org/jira/browse/LIVY-754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido resolved LIVY-754.
--
Fix Version/s: 0.8.0
Assignee: Wing Yew Poon
Resolution: Fixed
Issue resolved by PR: https://github.com/apache/incubator-livy/pull/288.
> precision and scale are not encoded in decimal type
> ---
>
> Key: LIVY-754
> URL: https://issues.apache.org/jira/browse/LIVY-754
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Affects Versions: 0.7.0
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
>Priority: Major
> Fix For: 0.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The Livy Thrift server support for decimal type in 0.7 is inadequate.
> Before LIVY-699, decimal is mapped to the catch-all string type. With
> LIVY-699, decimal is mapped to a decimal type that is inadequate in that it
> does not encode the precision and scale. The type in Livy is represented by a
> BasicDataType case class which contains a String field, name; and a DataType
> (an enum) field, dataType. In the case of decimal, the dataType is
> DataType.DECIMAL. The precision and scale of the decimal is not encoded.
> When the DataType is converted to a TTypeDesc for sending a Thrift response
> to a client request for result set metadata, the TTypeDesc contains a
> TPrimitiveTypeEntry(TTypeId.DECIMAL_TYPE) without TTypeQualifiers (which are
> needed to capture the precision and scale). This results in problems for
> clients. E.g., if we connect to the Thrift server in beeline and do a select
> from a table with column of decimal type, we get
> {noformat}
> java.lang.NullPointerException
> at org.apache.hive.jdbc.JdbcColumn.columnPrecision(JdbcColumn.java:310)
> at
> org.apache.hive.jdbc.JdbcColumn.columnDisplaySize(JdbcColumn.java:262)
> at
> org.apache.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(HiveResultSetMetaData.java:63)
> at
> org.apache.hive.beeline.IncrementalRows.(IncrementalRows.java:57)
> at
> org.apache.hive.beeline.IncrementalRowsWithNormalization.(IncrementalRowsWithNormalization.java:47)
> at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2322)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1026)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1215)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1144)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1134)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
> {noformat}
> Note: You have to use "--verbose" with beeline to see the stack trace for the
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (LIVY-752) Livy TS does not accept any connections when limits are set on connections
[
https://issues.apache.org/jira/browse/LIVY-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido resolved LIVY-752.
--
Fix Version/s: 0.8.0
Assignee: Wing Yew Poon
Resolution: Fixed
Issue resolved by PR: [https://github.com/apache/incubator-livy/pull/284].
> Livy TS does not accept any connections when limits are set on connections
> --
>
> Key: LIVY-752
> URL: https://issues.apache.org/jira/browse/LIVY-752
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Affects Versions: 0.7.0
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
>Priority: Major
> Fix For: 0.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I set livy.server.thrift.limit.connections.per.user=20 on my Livy Server.
> When I try to connect to it, I get
> {noformat}
> 2020-02-28 17:13:30,443 WARN
> org.apache.livy.thriftserver.cli.ThriftBinaryCLIService: Error opening
> session:
> java.lang.NullPointerException
> at
> org.apache.livy.thriftserver.LivyThriftSessionManager.incrementConnectionsCount(LivyThriftSessionManager.scala:438)
> at
> org.apache.livy.thriftserver.LivyThriftSessionManager.incrementConnections(LivyThriftSessionManager.scala:425)
> at
> org.apache.livy.thriftserver.LivyThriftSessionManager.openSession(LivyThriftSessionManager.scala:222)
> at
> org.apache.livy.thriftserver.LivyCLIService.openSessionWithImpersonation(LivyCLIService.scala:121)
> at
> org.apache.livy.thriftserver.cli.ThriftCLIService.getSessionHandle(ThriftCLIService.scala:324)
> at
> org.apache.livy.thriftserver.cli.ThriftCLIService.OpenSession(ThriftCLIService.scala:203)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (LIVY-745) more than 11 concurrent clients lead to java.io.IOException: Unable to connect to provided ports 10000~10010
[
https://issues.apache.org/jira/browse/LIVY-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido resolved LIVY-745.
--
Fix Version/s: 0.7.0
Assignee: Wing Yew Poon
Resolution: Fixed
Issue resolved by pull request
https://github.com/apache/incubator-livy/pull/275.
> more than 11 concurrent clients lead to java.io.IOException: Unable to
> connect to provided ports 1~10010
>
>
> Key: LIVY-745
> URL: https://issues.apache.org/jira/browse/LIVY-745
> Project: Livy
> Issue Type: Bug
> Components: RSC
>Affects Versions: 0.6.0
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
>Priority: Major
> Fix For: 0.7.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In testing scalability of the Livy Thrift server, I am simultaneously
> starting multiple connections to it. When there are more than 11 connections
> started simultaneously, the 12th (and subsequent) connection will fail with:
> {noformat}
> 2020-01-10 13:53:28,686 ERROR
> org.apache.livy.thriftserver.LivyExecuteStatementOperation: Error running
> hive query:
> org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException:
> java.io.IOException: Unable to connect to provided ports 1~10010
> {noformat}
> Here is the excerpt from the Livy server log:
> {noformat}
> 2020-01-10 13:53:28,138 INFO
> org.apache.livy.server.interactive.InteractiveSession$: Creating Interactive
> session 0: [owner: systest, request: [kind: spark, proxyUser: None,
> heartbeatTimeoutInSecond: 0]]
> ...
> 2020-01-10 13:53:28,147 INFO
> org.apache.livy.server.interactive.InteractiveSession$: Creating Interactive
> session 1: [owner: systest, request: [kind: spark, proxyUser: None,
> heartbeatTimeoutInSecond: 0]]
> ...
> 2020-01-10 13:53:28,196 INFO
> org.apache.livy.server.interactive.InteractiveSession$: Creating Interactive
> session 2: [owner: systest, request: [kind: spark, proxyUser: None,
> heartbeatTimeoutInSecond: 0]]
> ...
> 2020-01-10 13:53:28,247 INFO
> org.apache.livy.server.interactive.InteractiveSession$: Creating Interactive
> session 3: [owner: systest, request: [kind: spark, proxyUser: None,
> heartbeatTimeoutInSecond: 0]]
> ...
> 2020-01-10 13:53:28,304 INFO
> org.apache.livy.server.interactive.InteractiveSession$: Creating Interactive
> session 4: [owner: systest, request: [kind: spark, proxyUser: None,
> heartbeatTimeoutInSecond: 0]]
> 2020-01-10 13:53:28,329 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 1 Address already in use
> 2020-01-10 13:53:28,329 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 1 Address already in use
> 2020-01-10 13:53:28,329 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 1 Address already in use
> 2020-01-10 13:53:28,329 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 1 Address already in use
> 2020-01-10 13:53:28,329 INFO org.apache.livy.rsc.rpc.RpcServer: Connected to
> the port 1
> ...
> 2020-01-10 13:53:28,331 INFO org.apache.livy.rsc.rpc.RpcServer: Connected to
> the port 10001
> ...
> 2020-01-10 13:53:28,335 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10001 Address already in use
> 2020-01-10 13:53:28,335 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10001 Address already in use
> 2020-01-10 13:53:28,335 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10001 Address already in use
> 2020-01-10 13:53:28,338 INFO org.apache.livy.rsc.rpc.RpcServer: Connected to
> the port 10002
> 2020-01-10 13:53:28,338 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10002 Address already in use
> ...
> 2020-01-10 13:53:28,339 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10002 Address already in use
> 2020-01-10 13:53:28,341 INFO org.apache.livy.rsc.rpc.RpcServer: Connected to
> the port 10003
> ...
> 2020-01-10 13:53:28,341 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10003 Address already in use
> ...
> 2020-01-10 13:53:28,343 INFO org.apache.livy.rsc.rpc.RpcServer: Connected to
> the port 10004
> ...
> 2020-01-10 13:53:28,362 INFO
> org.apache.livy.server.interactive.InteractiveSession$: Creating Interactive
> session 5: [owner: systest, request: [kind: spark, proxyUser: None,
> heartbeatTimeoutInSecond: 0]]
> 2020-01-10 13:53:28,367 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 1 Address already in use
> 2020-01-10 13:53:28,369 DEBUG org.apache.livy.rsc.rpc.RpcServer: RPC not able
> to connect port 10001 Address already in use
> 2020-01-10 13:53:28,371 DEBUG
[jira] [Comment Edited] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005512#comment-17005512 ] Marco Gaido edited comment on LIVY-718 at 12/30/19 6:29 PM: bq. IIRC JDBC had a REST and an RPC mode. The RPC mode might not be HA without a fat client but perhaps the REST mode could. Does Hive JDBC support HA on the Hive Thrift server? Then maybe the hive JDBC client now supports server side transitions. If not, then we may have the caveat that HA won't work for such connections. I am not super familiar with the JDBC client. Actually, JDBC has either HTTP or binary mode, but it relates only to the communication between the Livy server and the client and it is unrelated to how the Livy server interacts with Spark (ie. via RPC). was (Author: mgaido): > IIRC JDBC had a REST and an RPC mode. The RPC mode might not be HA without a > fat client but perhaps the REST mode could. Does Hive JDBC support HA on the > Hive Thrift server? Then maybe the hive JDBC client now supports server side > transitions. If not, then we may have the caveat that HA won't work for such > connections. I am not super familiar with the JDBC client. Actually, JDBC has either HTTP or binary mode, but it relates only to the communication between the Livy server and the client and it is unrelated to how the Livy server interacts with Spark (ie. via RPC). > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005512#comment-17005512 ] Marco Gaido commented on LIVY-718: -- > IIRC JDBC had a REST and an RPC mode. The RPC mode might not be HA without a > fat client but perhaps the REST mode could. Does Hive JDBC support HA on the > Hive Thrift server? Then maybe the hive JDBC client now supports server side > transitions. If not, then we may have the caveat that HA won't work for such > connections. I am not super familiar with the JDBC client. Actually, JDBC has either HTTP or binary mode, but it relates only to the communication between the Livy server and the client and it is unrelated to how the Livy server interacts with Spark (ie. via RPC). > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005063#comment-17005063 ] Marco Gaido commented on LIVY-718: -- Thanks for your comments [~bikassaha]. I am not sure (actually I think it is not true) that all the metadata information is present in the Spark driver process and there is metadata which is frequently accessed/changed (eg. the ongoing statements for a given session) on the Livy server side (at least for the thrift part). Indeed, there are definitely metadata which are currently kept in the server memory which would need to be saved for HA sake. Hence, I am afraid that at least for the thrift case, the usage of a slow storage like HDFS would at least would require a significant revisit of the thrift part. I agree that active-active is by far the most desirable choice. I see, though, that it is not easy to implement, IMHO, because for the metadata above, it would require a distributed state store being the source of truth for that. Given your negative opinion on ZK, I hardly see any other system which would fit (a relational DB cluster maybe? but not easier to maintain than ZK for sure, I'd say). Hence I am drawn to consider that we would need to trade off things here, unless I am very mistaken on the point above: namely, the REST part has really no significant metadata on Livy server side and we keep the thrift one out of scope here. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (LIVY-705) Support getting keystore password from Hadoop credential provider for Thrift server
[ https://issues.apache.org/jira/browse/LIVY-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-705: Assignee: Wing Yew Poon > Support getting keystore password from Hadoop credential provider for Thrift > server > --- > > Key: LIVY-705 > URL: https://issues.apache.org/jira/browse/LIVY-705 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Affects Versions: 0.6.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 0.7.0 > > Time Spent: 20m > Remaining Estimate: 0h > > LIVY-475 added support for getting the keystore password and key password > from a Hadoop credential provider file. The keystore password is also needed > for SSL/TLS support in the Thrift server. We should extend the support for > getting the keystore password from the Hadoop credential provider to the > Thrift server as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (LIVY-705) Support getting keystore password from Hadoop credential provider for Thrift server
[ https://issues.apache.org/jira/browse/LIVY-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved LIVY-705. -- Fix Version/s: 0.7.0 Resolution: Fixed Issue resolved by PR https://github.com/apache/incubator-livy/pull/253. > Support getting keystore password from Hadoop credential provider for Thrift > server > --- > > Key: LIVY-705 > URL: https://issues.apache.org/jira/browse/LIVY-705 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Affects Versions: 0.6.0 >Reporter: Wing Yew Poon >Priority: Major > Fix For: 0.7.0 > > Time Spent: 20m > Remaining Estimate: 0h > > LIVY-475 added support for getting the keystore password and key password > from a Hadoop credential provider file. The keystore password is also needed > for SSL/TLS support in the Thrift server. We should extend the support for > getting the keystore password from the Hadoop credential provider to the > Thrift server as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (LIVY-356) Add support for LDAP authentication in Livy Server
[ https://issues.apache.org/jira/browse/LIVY-356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved LIVY-356. -- Fix Version/s: 0.7.0 Assignee: mingchao zhao Resolution: Fixed Issue resolved by https://github.com/apache/incubator-livy/pull/231. > Add support for LDAP authentication in Livy Server > -- > > Key: LIVY-356 > URL: https://issues.apache.org/jira/browse/LIVY-356 > Project: Livy > Issue Type: New Feature > Components: Server >Affects Versions: 0.4.0 >Reporter: Janki Akhani >Assignee: mingchao zhao >Priority: Major > Fix For: 0.7.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, Livy doesn't support LDAP Authentication from client(sparkmagic) > to server(livy). We need to add LDAP authentication as that's preferable > method due to security reasons. We won't be able to use Knox for this > purpose. That is why I am raising this PR which contains LDAP authentication. > I have upgraded hadoop.version in livy-main to 2.8.0 as this version contains > LivyAuthenticationHandler. Below I have mentioned link for the same: > https://insight.io/github.com/apache/hadoop/blob/HEAD/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-667) Support query a lot of data.
[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935852#comment-16935852 ] Marco Gaido commented on LIVY-667: -- [~yihengw] yes, that's true. The point is that the data needs to be transferred anyway through JDBC. So having very very large datasets going over the wire may not very efficient. Moreover, in case you have a single very big partition, you can always repartition it and avoid the issue. My point here is that there may be workarounds for the use case and I don't expect this problem to be faced in usual use cases. So I feel an overkill to design some workarounds for a corner case. It is also doable to do the same which is suggested here manually: ie. create a table with the result of a query (this writes on HDFS) and then read the table... > Support query a lot of data. > > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver >Affects Versions: 0.6.0 >Reporter: runzhiwang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-667) Support query a lot of data.
[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935616#comment-16935616 ] Marco Gaido commented on LIVY-667: -- [~runzhiwang] let me cite you in the JIRA body: > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time this means that if the driver is as big as the executors, there should not be OOM, since the partition was already held in memory on the executors. And in general having a driver large as the executors should not be a big deal in terms on memory occupation on the cluster in percentage, as I expect to have much more executors than drivers.. > Support query a lot of data. > > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver >Affects Versions: 0.6.0 >Reporter: runzhiwang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (LIVY-683) Livy SQLInterpreter get empty array when extract date format rows to json
[
https://issues.apache.org/jira/browse/LIVY-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido resolved LIVY-683.
--
Resolution: Duplicate
> Livy SQLInterpreter get empty array when extract date format rows to json
> -
>
> Key: LIVY-683
> URL: https://issues.apache.org/jira/browse/LIVY-683
> Project: Livy
> Issue Type: Bug
> Components: REPL
>Reporter: Zhefeng Wang
>Priority: Major
> Attachments: image-2019-09-17-14-23-31-715.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I submit a sql with select date format:
> {code:java}
> select to_date(update_time) from sec_ods.ods_binlog_task_list_d_whole where
> concat(YEAR, MONTH, DAY )=20190306 limit 10
> {code}
> in livy:
> !image-2019-09-17-14-23-31-715.png|width=429,height=424!
> we can see data is composed by some empty arrays.
> and in spark this sql can get correct result:
> {code:java}
> to_date(sec_ods.ods_binlog_task_list_d_whole.`update_time`)
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (LIVY-683) Livy SQLInterpreter get empty array when extract date format rows to json
[
https://issues.apache.org/jira/browse/LIVY-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido reassigned LIVY-683:
Assignee: (was: Zhefeng Wang)
> Livy SQLInterpreter get empty array when extract date format rows to json
> -
>
> Key: LIVY-683
> URL: https://issues.apache.org/jira/browse/LIVY-683
> Project: Livy
> Issue Type: Bug
> Components: REPL
>Reporter: Zhefeng Wang
>Priority: Major
> Attachments: image-2019-09-17-14-23-31-715.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I submit a sql with select date format:
> {code:java}
> select to_date(update_time) from sec_ods.ods_binlog_task_list_d_whole where
> concat(YEAR, MONTH, DAY )=20190306 limit 10
> {code}
> in livy:
> !image-2019-09-17-14-23-31-715.png|width=429,height=424!
> we can see data is composed by some empty arrays.
> and in spark this sql can get correct result:
> {code:java}
> to_date(sec_ods.ods_binlog_task_list_d_whole.`update_time`)
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> 2019-03-04
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-667) Support query a lot of data.
[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934489#comment-16934489 ] Marco Gaido commented on LIVY-667: -- [~runzhiwang] I'd argue that if the data which is present in a single partition doesn't fit in the driver, you just need to increase the memory on the driver... > Support query a lot of data. > > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver >Affects Versions: 0.6.0 >Reporter: runzhiwang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924894#comment-16924894 ] Marco Gaido commented on LIVY-489: -- The right place is the user mailing list, please see https://livy.apache.org/community/. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924817#comment-16924817 ] Marco Gaido commented on LIVY-489: -- Yes, sure. The point is: that template in not exhaustive and does not contain all the configs which can be set. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[
https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924812#comment-16924812
]
Marco Gaido commented on LIVY-489:
--
[~vonatzki] nothing to be sorry about. Please see
https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/LivyConf.scala#L101.
{{livy.server.thrift.enabled}} must be set to true, the port is controlled by
https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/LivyConf.scala#L109.
Then you should be able to connect using the beeline you can find in the repo
or with any Hive 3 client. Thanks.
> Expose a JDBC endpoint for Livy
> ---
>
> Key: LIVY-489
> URL: https://issues.apache.org/jira/browse/LIVY-489
> Project: Livy
> Issue Type: New Feature
> Components: API, Server
>Affects Versions: 0.6.0
>Reporter: Marco Gaido
>Assignee: Marco Gaido
>Priority: Major
> Fix For: 0.6.0
>
>
> Many users and BI tools use JDBC connections in order to retrieve data. As
> Livy exposes only a REST API, this is a limitation in its adoption. Hence,
> adding a JDBC endpoint may be a very useful feature, which could also make
> Livy a more attractive solution for end user to adopt.
> Moreover, currently, Spark exposes a JDBC interface, but this has many
> limitations, including that all the queries are submitted to the same
> application, therefore there is no isolation/security, which can be offered
> by Livy, making a Livy JDBC API a better solution for companies/users who
> want to use Spark in order to run they queries through JDBC.
> In order to make the transition from existing solutions to the new JDBC
> server seamless, the proposal is to use the Hive thrift-server and extend it
> as it was done by the STS.
> [Here, you can find the design
> doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922363#comment-16922363 ] Marco Gaido commented on LIVY-489: -- Hi [~vonatzki]. Please check the design doc attached here. Moreover, please check the configurations added in LivyConf. You'll see that the thriftserver must be enabled and configured properly and default values for the configurations are there. As a suggestion, it is possible for you, I'd recommend to build Livy from master branch, even though the thriftserver is present also in the 0.6.0 version. Indeed, it was the first release for it, so might find issues which have been fixed on current master. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (LIVY-650) Remove schema from ResultSet
[
https://issues.apache.org/jira/browse/LIVY-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido resolved LIVY-650.
--
Fix Version/s: 0.7.0
Resolution: Fixed
Issue resolved by https://github.com/apache/incubator-livy/pull/213.
> Remove schema from ResultSet
>
>
> Key: LIVY-650
> URL: https://issues.apache.org/jira/browse/LIVY-650
> Project: Livy
> Issue Type: Improvement
> Components: Thriftserver
>Affects Versions: 0.7.0
>Reporter: Marco Gaido
>Assignee: Marco Gaido
>Priority: Minor
> Fix For: 0.7.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{ResultSet}} class currently contains the schema despite it is never
> used.
> We can get rid of it and this is a benefit because we don't serialize this
> string which may be pretty big.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (LIVY-650) Remove schema from ResultSet
Marco Gaido created LIVY-650:
Summary: Remove schema from ResultSet
Key: LIVY-650
URL: https://issues.apache.org/jira/browse/LIVY-650
Project: Livy
Issue Type: Improvement
Components: Thriftserver
Affects Versions: 0.7.0
Reporter: Marco Gaido
Assignee: Marco Gaido
The {{ResultSet}} class currently contains the schema despite it is never used.
We can get rid of it and this is a benefit because we don't serialize this
string which may be pretty big.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (LIVY-573) Add tests for operation logs retrieval
[ https://issues.apache.org/jira/browse/LIVY-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-573: Assignee: Yiheng Wang > Add tests for operation logs retrieval > -- > > Key: LIVY-573 > URL: https://issues.apache.org/jira/browse/LIVY-573 > Project: Livy > Issue Type: Improvement > Components: Tests, Thriftserver >Reporter: Marco Gaido >Assignee: Yiheng Wang >Priority: Trivial > Time Spent: 1h > Remaining Estimate: 0h > > Our current tests do not cover the retrieval of operation logs. We should try > and add coverage for it if possible. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-627) Implement GetCrossReference metadata operation
[ https://issues.apache.org/jira/browse/LIVY-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902733#comment-16902733 ] Marco Gaido commented on LIVY-627: -- Hi [~micahzhao]. Actually in Spark there is no primary key/foreign key at all. So there is few we can do here I think. Those operations are not supported by Spark itself.. > Implement GetCrossReference metadata operation > -- > > Key: LIVY-627 > URL: https://issues.apache.org/jira/browse/LIVY-627 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetCrossReference metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900970#comment-16900970 ] Marco Gaido commented on LIVY-575: -- No problem, thank you. :) > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-627) Implement GetCrossReference metadata operation
[ https://issues.apache.org/jira/browse/LIVY-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-627: Assignee: (was: Yiheng Wang) > Implement GetCrossReference metadata operation > -- > > Key: LIVY-627 > URL: https://issues.apache.org/jira/browse/LIVY-627 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetCrossReference metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900956#comment-16900956 ] Marco Gaido edited comment on LIVY-575 at 8/6/19 12:12 PM: --- [~yihengw] we assign JIRAs when the related PRs are merged. You can rather leave a comment on them if you plan to work on them so noone else will spend time on them. Thanks. was (Author: mgaido): [~yihengw] we assign JIRAs when the related PRs are merged. Thanks. > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-628) Implement GetDelegationToken metadata operation
[ https://issues.apache.org/jira/browse/LIVY-628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-628: Assignee: (was: Yiheng Wang) > Implement GetDelegationToken metadata operation > --- > > Key: LIVY-628 > URL: https://issues.apache.org/jira/browse/LIVY-628 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetDelegationToken metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-626) Implement GetPrimaryKeys metadata operation
[ https://issues.apache.org/jira/browse/LIVY-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-626: Assignee: (was: Yiheng Wang) > Implement GetPrimaryKeys metadata operation > --- > > Key: LIVY-626 > URL: https://issues.apache.org/jira/browse/LIVY-626 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetPrimaryKeys metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-623) Implement GetTables metadata operation
[ https://issues.apache.org/jira/browse/LIVY-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-623: Assignee: (was: Yiheng Wang) > Implement GetTables metadata operation > -- > > Key: LIVY-623 > URL: https://issues.apache.org/jira/browse/LIVY-623 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetTables metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-624) Implement GetColumns metadata operation
[ https://issues.apache.org/jira/browse/LIVY-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-624: Assignee: (was: Yiheng Wang) > Implement GetColumns metadata operation > --- > > Key: LIVY-624 > URL: https://issues.apache.org/jira/browse/LIVY-624 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetColumns metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-625) Implement GetFunctions metadata operation
[ https://issues.apache.org/jira/browse/LIVY-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-625: Assignee: (was: Yiheng Wang) > Implement GetFunctions metadata operation > - > > Key: LIVY-625 > URL: https://issues.apache.org/jira/browse/LIVY-625 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Priority: Minor > > We should support GetFunctions metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-575: Assignee: (was: Yiheng Wang) > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900956#comment-16900956 ] Marco Gaido commented on LIVY-575: -- [~yihengw] we assign JIRAs when the related PRs are merged. Thanks. > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890784#comment-16890784 ] Marco Gaido commented on LIVY-575: -- Oh, you're right. Sorry for the mistake. > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-571) When an exception happens running a query, we should report it to end user
[
https://issues.apache.org/jira/browse/LIVY-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890785#comment-16890785
]
Marco Gaido commented on LIVY-571:
--
Issue resolved by PR https://github.com/apache/incubator-livy/pull/182.
> When an exception happens running a query, we should report it to end user
> --
>
> Key: LIVY-571
> URL: https://issues.apache.org/jira/browse/LIVY-571
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Reporter: Marco Gaido
>Assignee: Jeffrey(Xilang) Yan
>Priority: Major
> Fix For: 0.7.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When a query fails with an exception on livy thriftserver, instead of
> reporting this exception to the end user, a meaningless one is reported. Eg,
> with Hive support not enabled on spark, the following query causes:
> {code}
> 0: jdbc:hive2://localhost:10090/> create table test as select a.* from
> (select 1, "2") a;
> Error: java.lang.RuntimeException: java.util.NoSuchElementException:
> Statement 820bb5c2-018b-46ea-9b7f-b0e3b9c31c46 not found in session
> acf3712b-1f08-4111-950f-559fc3f3f10c.
> org.apache.livy.thriftserver.session.ThriftSessionState.statementNotFound(ThriftSessionState.java:118)
> org.apache.livy.thriftserver.session.ThriftSessionState.cleanupStatement(ThriftSessionState.java:107)
> org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:43)
> org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:26)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
> Looking at the logs, of course the real problem is:
> {code}
> 19/03/23 10:40:32 ERROR LivyExecuteStatementOperation: Error running hive
> query:
> org.apache.hive.service.cli.HiveSQLException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.spark.sql.AnalysisException: Hive support is required to CREATE
> Hive TABLE (AS SELECT);;
> 'CreateTable `test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> ErrorIfExists
> +- Project [1#1, 2#2]
>+- SubqueryAlias `a`
> +- Project [1 AS 1#1, 2 AS 2#2]
> +- OneRowRelation
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:392)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:390)
> org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:117)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:390)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:388)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:386)
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
> org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
> org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
> org.apache.livy.thriftserver.session.SqlJob.executeSql(SqlJob.java:74)
> org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:64)
> org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:35)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
[jira] [Issue Comment Deleted] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated LIVY-575: - Comment: was deleted (was: Issue resolved by PR https://github.com/apache/incubator-livy/pull/182.) > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-571) When an exception happens running a query, we should report it to end user
[
https://issues.apache.org/jira/browse/LIVY-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido reassigned LIVY-571:
Assignee: Jeffrey(Xilang) Yan
> When an exception happens running a query, we should report it to end user
> --
>
> Key: LIVY-571
> URL: https://issues.apache.org/jira/browse/LIVY-571
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Reporter: Marco Gaido
>Assignee: Jeffrey(Xilang) Yan
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When a query fails with an exception on livy thriftserver, instead of
> reporting this exception to the end user, a meaningless one is reported. Eg,
> with Hive support not enabled on spark, the following query causes:
> {code}
> 0: jdbc:hive2://localhost:10090/> create table test as select a.* from
> (select 1, "2") a;
> Error: java.lang.RuntimeException: java.util.NoSuchElementException:
> Statement 820bb5c2-018b-46ea-9b7f-b0e3b9c31c46 not found in session
> acf3712b-1f08-4111-950f-559fc3f3f10c.
> org.apache.livy.thriftserver.session.ThriftSessionState.statementNotFound(ThriftSessionState.java:118)
> org.apache.livy.thriftserver.session.ThriftSessionState.cleanupStatement(ThriftSessionState.java:107)
> org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:43)
> org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:26)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
> Looking at the logs, of course the real problem is:
> {code}
> 19/03/23 10:40:32 ERROR LivyExecuteStatementOperation: Error running hive
> query:
> org.apache.hive.service.cli.HiveSQLException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.spark.sql.AnalysisException: Hive support is required to CREATE
> Hive TABLE (AS SELECT);;
> 'CreateTable `test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> ErrorIfExists
> +- Project [1#1, 2#2]
>+- SubqueryAlias `a`
> +- Project [1 AS 1#1, 2 AS 2#2]
> +- OneRowRelation
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:392)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:390)
> org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:117)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:390)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:388)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:386)
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
> org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
> org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
> org.apache.livy.thriftserver.session.SqlJob.executeSql(SqlJob.java:74)
> org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:64)
> org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:35)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
[jira] [Resolved] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved LIVY-575. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by PR https://github.com/apache/incubator-livy/pull/182. > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > Fix For: 0.7.0 > > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (LIVY-598) Upgrade Livy jackson dependency to 2.9.9
[ https://issues.apache.org/jira/browse/LIVY-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved LIVY-598. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by https://github.com/apache/incubator-livy/pull/174. > Upgrade Livy jackson dependency to 2.9.9 > > > Key: LIVY-598 > URL: https://issues.apache.org/jira/browse/LIVY-598 > Project: Livy > Issue Type: Improvement >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan >Priority: Major > Fix For: 0.7.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Upgrade the jackson dependency to 2.9.9 which fixes CVE-2019-12086. > Spark has also recently upgraded to jackson version 2.9.9. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-598) Upgrade Livy jackson dependency to 2.9.9
[ https://issues.apache.org/jira/browse/LIVY-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-598: Assignee: Arun Mahadevan > Upgrade Livy jackson dependency to 2.9.9 > > > Key: LIVY-598 > URL: https://issues.apache.org/jira/browse/LIVY-598 > Project: Livy > Issue Type: Improvement >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Upgrade the jackson dependency to 2.9.9 which fixes CVE-2019-12086. > Spark has also recently upgraded to jackson version 2.9.9. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887734#comment-16887734 ] Marco Gaido commented on LIVY-575: -- HI [~yihengw]. Thanks for your help on this. I think it would be great to have them in a single PR. Looking forward to it! Thanks. > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-590) ClassNotFoundException: javax.ws.rs.ext.MessageBodyReader on Livy 0.6.0
[
https://issues.apache.org/jira/browse/LIVY-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824936#comment-16824936
]
Marco Gaido commented on LIVY-590:
--
Hi [~tanakahda], thanks for reporting this. Livy works with PRs and not with
patches (please see
[https://livy.incubator.apache.org/community/#contributing]). May you please
create a PR? We can discuss the details of the issue there.
Moreover, it would be great if you could explain in the description of the PR
you're going to open why this is happening on your env, but the CI and also
other environments don't have this issue.
Thanks.
> ClassNotFoundException: javax.ws.rs.ext.MessageBodyReader on Livy 0.6.0
> ---
>
> Key: LIVY-590
> URL: https://issues.apache.org/jira/browse/LIVY-590
> Project: Livy
> Issue Type: Bug
> Components: Server
>Affects Versions: 0.6.0
>Reporter: Aki Tanaka
>Priority: Major
> Attachments: LIVY-590.patch
>
>
> After I upgraded Livy to 0.6.0-incubating, running Spark job using Livy
> started failing. The details of the problem are below.
> 1. When I start Livy server, following message is logged:
> {quote}19/04/18 23:13:35 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> java.lang.NoClassDefFoundError: javax/ws/rs/ext/MessageBodyReader
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> ..
> Caused by: java.lang.ClassNotFoundException:
> javax.ws.rs.ext.MessageBodyReader
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 50 more
> {quote}
>
> 2. When I submit a Spark job using Livy, the job state stuck on "starting".
> Also Livy cannot get the job's appId.
> {quote}$ curl [http://10.10.144.20:8998/batches]
> \{ "from": 0, "total": 1, "sessions": [ \{ "id": 0, "name": null, "state":
> "starting", "appId": null, "appInfo": \{ "driverLogUrl": null, "sparkUiUrl":
> null \}, "log": [ "19/04/18 20:28:58 INFO MemoryStore: MemoryStore cleared",
> "19/04/18 20:28:58 INFO BlockManager: BlockManager stopped", "19/04/18
> 20:28:58 INFO BlockManagerMaster: BlockManagerMaster stopped", "19/04/18
> 20:28:58 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!", "19/04/18 20:28:58 INFO SparkContext:
> Successfully stopped SparkContext", "19/04/18 20:28:58 INFO
> ShutdownHookManager: Shutdown hook called", "19/04/18 20:28:58 INFO
> ShutdownHookManager: Deleting directory
> /mnt/tmp/spark-b8039adb-f3df-4526-8123-1bb2aee6ed7c", "19/04/18 20:28:58 INFO
> ShutdownHookManager: Deleting directory
> /mnt/tmp/spark-b48219fd-2607-4a7d-95bd-538c89f90ebb", "\nstderr: ", "\nYARN
> Diagnostics: " ] \} ] \}
> {quote}
>
> This is caused because Livy package does not have jersey-core jar file. This
> change was introduced by LIVY-502
> I think Livy package should have the jersey-core jar file. Modifying
> server/pom.xml (I attached the diff to this JIRA) was able to run Spark jobs
> without this error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (LIVY-572) Log operations fails because of dependencies on Spark
[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido reassigned LIVY-572:
Assignee: Marco Gaido
> Log operations fails because of dependencies on Spark
> -
>
> Key: LIVY-572
> URL: https://issues.apache.org/jira/browse/LIVY-572
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: Marco Gaido
>Assignee: Marco Gaido
>Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is
> by default, all queries run through beeline fail due to:
> {code}
> 19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
> SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
> Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
> java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
> at
> org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
> at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
> at
> org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
> at
> org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
> at
> org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
> at
> org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LIVY-572) Log operations fails because of dependencies on Spark
[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-572:
-
Description:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, all queries run through beeline fail due to:
{code}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
{code}
was:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all queries run through
beeline fail due to:
{code}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
[jira] [Updated] (LIVY-572) Log operations fails because of dependencies on Spark
[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-572:
-
Description:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all queries run through
beeline fail due to:
{code}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
{code}
was:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all operations fail due
to:
{code}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at
[jira] [Updated] (LIVY-572) Log operations fails because of dependencies on Spark
[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-572:
-
Priority: Critical (was: Major)
> Log operations fails because of dependencies on Spark
> -
>
> Key: LIVY-572
> URL: https://issues.apache.org/jira/browse/LIVY-572
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Reporter: Marco Gaido
>Priority: Critical
>
> When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is
> by default, and Spark jars are not available in Livy JVM, all operations fail
> due to:
> {code}
> 19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
> SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
> Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
> java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
> at
> org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
> at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
> at
> org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
> at
> org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
> at
> org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
> at
> org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LIVY-572) Log operations fails because of dependencies on Spark
[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-572:
-
Affects Version/s: 0.6.0
> Log operations fails because of dependencies on Spark
> -
>
> Key: LIVY-572
> URL: https://issues.apache.org/jira/browse/LIVY-572
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: Marco Gaido
>Priority: Critical
>
> When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is
> by default, and Spark jars are not available in Livy JVM, all operations fail
> due to:
> {code}
> 19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
> SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
> Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
> java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
> at
> org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
> at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
> at
> org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
> at
> org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
> at
> org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
> at
> org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LIVY-574) Add tests for metadata operations
[ https://issues.apache.org/jira/browse/LIVY-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated LIVY-574: - Priority: Major (was: Trivial) > Add tests for metadata operations > - > > Key: LIVY-574 > URL: https://issues.apache.org/jira/browse/LIVY-574 > Project: Livy > Issue Type: Improvement > Components: Tests, Thriftserver >Reporter: Marco Gaido >Priority: Major > > We do not have tests for metadata operations. We should add them at least for > the ones which we have currently implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-574) Add tests for metadata operations
Marco Gaido created LIVY-574: Summary: Add tests for metadata operations Key: LIVY-574 URL: https://issues.apache.org/jira/browse/LIVY-574 Project: Livy Issue Type: Improvement Components: Tests, Thriftserver Reporter: Marco Gaido We do not have tests for metadata operations. We should add them at least for the ones which we have currently implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-573) Add tests for operation logs retrieval
Marco Gaido created LIVY-573: Summary: Add tests for operation logs retrieval Key: LIVY-573 URL: https://issues.apache.org/jira/browse/LIVY-573 Project: Livy Issue Type: Improvement Components: Tests, Thriftserver Reporter: Marco Gaido Our current tests do not cover the retrieval of operation logs. We should try and add coverage for it if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LIVY-572) Log operations fails because of dependencies on Spark
[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-572:
-
Description:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all operations fail due
to:
{code}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
{code}
was:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all operations fail due
to:
{{code}}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[jira] [Created] (LIVY-572) Log operations fails because of dependencies on Spark
Marco Gaido created LIVY-572:
Summary: Log operations fails because of dependencies on Spark
Key: LIVY-572
URL: https://issues.apache.org/jira/browse/LIVY-572
Project: Livy
Issue Type: Bug
Components: Thriftserver
Reporter: Marco Gaido
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all operations fail due
to:
{{code}}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
{{code}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LIVY-571) When an exception happens running a query, we should report it to end user
[
https://issues.apache.org/jira/browse/LIVY-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-571:
-
Component/s: Thriftserver
> When an exception happens running a query, we should report it to end user
> --
>
> Key: LIVY-571
> URL: https://issues.apache.org/jira/browse/LIVY-571
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
>Reporter: Marco Gaido
>Priority: Major
>
> When a query fails with an exception on livy thriftserver, instead of
> reporting this exception to the end user, a meaningless one is reported. Eg,
> with Hive support not enabled on spark, the following query causes:
> {code}
> 0: jdbc:hive2://localhost:10090/> create table test as select a.* from
> (select 1, "2") a;
> Error: java.lang.RuntimeException: java.util.NoSuchElementException:
> Statement 820bb5c2-018b-46ea-9b7f-b0e3b9c31c46 not found in session
> acf3712b-1f08-4111-950f-559fc3f3f10c.
> org.apache.livy.thriftserver.session.ThriftSessionState.statementNotFound(ThriftSessionState.java:118)
> org.apache.livy.thriftserver.session.ThriftSessionState.cleanupStatement(ThriftSessionState.java:107)
> org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:43)
> org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:26)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
> Looking at the logs, of course the real problem is:
> {code}
> 19/03/23 10:40:32 ERROR LivyExecuteStatementOperation: Error running hive
> query:
> org.apache.hive.service.cli.HiveSQLException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.spark.sql.AnalysisException: Hive support is required to CREATE
> Hive TABLE (AS SELECT);;
> 'CreateTable `test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> ErrorIfExists
> +- Project [1#1, 2#2]
>+- SubqueryAlias `a`
> +- Project [1 AS 1#1, 2 AS 2#2]
> +- OneRowRelation
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:392)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:390)
> org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:117)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:390)
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:388)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:386)
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
> org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
> org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
> org.apache.livy.thriftserver.session.SqlJob.executeSql(SqlJob.java:74)
> org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:64)
> org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:35)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
> org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> at
>
[jira] [Commented] (LIVY-508) Support custom auth filter for LivyServer
[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765246#comment-16765246 ] Marco Gaido commented on LIVY-508: -- Thanks [~vanzin] > Support custom auth filter for LivyServer > - > > Key: LIVY-508 > URL: https://issues.apache.org/jira/browse/LIVY-508 > Project: Livy > Issue Type: Improvement > Components: Server >Affects Versions: 0.5.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Major > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Current Livy only supports Spnego auth support, we may have different > requirements for auth filter. So here propose to make Livy to support custom > auth filter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-508) Support custom auth filter for LivyServer
[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764863#comment-16764863 ] Marco Gaido commented on LIVY-508: -- Issue resolved by PR 100 https://github.com/apache/incubator-livy/pull/110 > Support custom auth filter for LivyServer > - > > Key: LIVY-508 > URL: https://issues.apache.org/jira/browse/LIVY-508 > Project: Livy > Issue Type: Improvement > Components: Server >Affects Versions: 0.5.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Major > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Current Livy only supports Spnego auth support, we may have different > requirements for auth filter. So here propose to make Livy to support custom > auth filter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-508) Support custom auth filter for LivyServer
[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764862#comment-16764862 ] Marco Gaido commented on LIVY-508: -- [~jerryshao] Sorry, seems like I don't have permissions to close the JIRA, may you please close this as "Resolved"? Thanks. > Support custom auth filter for LivyServer > - > > Key: LIVY-508 > URL: https://issues.apache.org/jira/browse/LIVY-508 > Project: Livy > Issue Type: Improvement > Components: Server >Affects Versions: 0.5.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Major > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Current Livy only supports Spnego auth support, we may have different > requirements for auth filter. So here propose to make Livy to support custom > auth filter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LIVY-508) Support custom auth filter for LivyServer
[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated LIVY-508: - Fix Version/s: 0.6.0 > Support custom auth filter for LivyServer > - > > Key: LIVY-508 > URL: https://issues.apache.org/jira/browse/LIVY-508 > Project: Livy > Issue Type: Improvement > Components: Server >Affects Versions: 0.5.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Major > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Current Livy only supports Spnego auth support, we may have different > requirements for auth filter. So here propose to make Livy to support custom > auth filter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-543) Fix inconsistent Jetty dependencies
Marco Gaido created LIVY-543: Summary: Fix inconsistent Jetty dependencies Key: LIVY-543 URL: https://issues.apache.org/jira/browse/LIVY-543 Project: Livy Issue Type: Sub-task Components: Server Reporter: Marco Gaido LIVY-526 introduced a new version for jetty which is different from the one used by Hive. This can cause conflicts in the version used and cause problems. Since the reason of LIVY-526 was to have security fixes, rather than reverting it we should try and have our version of jetty independent from the one used by Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (LIVY-538) Add integration tests for the thrift server
[ https://issues.apache.org/jira/browse/LIVY-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-538: Assignee: Marco Gaido > Add integration tests for the thrift server > --- > > Key: LIVY-538 > URL: https://issues.apache.org/jira/browse/LIVY-538 > Project: Livy > Issue Type: Sub-task > Components: Server, Tests >Affects Versions: 0.6.0 >Reporter: Marcelo Vanzin >Assignee: Marco Gaido >Priority: Major > > Currently there are no integration tests for the thrift server part. The UTs > are run on a different environment than ITs, which are more similar to a real > cluster, and having ITs would allow us to catch issues in the code that are > masked in UTs, such as serialization issues or referencing unavailable > classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-538) Add integration tests for the thrift server
[ https://issues.apache.org/jira/browse/LIVY-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707357#comment-16707357 ] Marco Gaido commented on LIVY-538: -- Created PR: https://github.com/apache/incubator-livy/pull/131. > Add integration tests for the thrift server > --- > > Key: LIVY-538 > URL: https://issues.apache.org/jira/browse/LIVY-538 > Project: Livy > Issue Type: Sub-task > Components: Server, Tests >Affects Versions: 0.6.0 >Reporter: Marcelo Vanzin >Assignee: Marco Gaido >Priority: Major > > Currently there are no integration tests for the thrift server part. The UTs > are run on a different environment than ITs, which are more similar to a real > cluster, and having ITs would allow us to catch issues in the code that are > masked in UTs, such as serialization issues or referencing unavailable > classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (LIVY-537) Add numExecutor configuration when creating session in thriftserver
[ https://issues.apache.org/jira/browse/LIVY-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-537: Assignee: Marco Gaido > Add numExecutor configuration when creating session in thriftserver > --- > > Key: LIVY-537 > URL: https://issues.apache.org/jira/browse/LIVY-537 > Project: Livy > Issue Type: Bug > Components: Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > > When parsing the option provided by the user in order to create a new session > using the thiftserver API, we are not setting the number of executors, which > is a very important parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-537) Add numExecutor configuration when creating session in thriftserver
[ https://issues.apache.org/jira/browse/LIVY-537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700423#comment-16700423 ] Marco Gaido commented on LIVY-537: -- Created PR: https://github.com/apache/incubator-livy/pull/130. > Add numExecutor configuration when creating session in thriftserver > --- > > Key: LIVY-537 > URL: https://issues.apache.org/jira/browse/LIVY-537 > Project: Livy > Issue Type: Bug > Components: Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > > When parsing the option provided by the user in order to create a new session > using the thiftserver API, we are not setting the number of executors, which > is a very important parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-537) Add numExecutor configuration when creating session in thriftserver
Marco Gaido created LIVY-537: Summary: Add numExecutor configuration when creating session in thriftserver Key: LIVY-537 URL: https://issues.apache.org/jira/browse/LIVY-537 Project: Livy Issue Type: Bug Components: Server Affects Versions: 0.6.0 Reporter: Marco Gaido When parsing the option provided by the user in order to create a new session using the thiftserver API, we are not setting the number of executors, which is a very important parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-520) Log when a session is deleted because of timeout
Marco Gaido created LIVY-520:
Summary: Log when a session is deleted because of timeout
Key: LIVY-520
URL: https://issues.apache.org/jira/browse/LIVY-520
Project: Livy
Issue Type: Improvement
Components: Server
Affects Versions: 0.6.0
Reporter: Marco Gaido
When a session is idle for more than {{livy.server.session.timeout}} it is
GCed, ie. it is closed. But there is no evidence that this is the reason why
the session is closed, so it may be non-trivial to detect the reason why the
session was closed.
This JIRA proposes to add a log information when a session is GCed in order to
make it clearer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (LIVY-506) Dedicated thread for timeout checker
Marco Gaido created LIVY-506: Summary: Dedicated thread for timeout checker Key: LIVY-506 URL: https://issues.apache.org/jira/browse/LIVY-506 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido The timeout checker task currently uses the backgroud pool of threads. Since the thread is always alive, this doesn't make a lot of sense as it is using forever a thread from the pool. It should use, instead, its dedicated thread. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-503) More RPC classes used in thrifserver in a separate module
Marco Gaido created LIVY-503: Summary: More RPC classes used in thrifserver in a separate module Key: LIVY-503 URL: https://issues.apache.org/jira/browse/LIVY-503 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido As suggested in the discussion for the original PR (https://github.com/apache/incubator-livy/pull/104#discussion_r212806490), we should move the RPC classes which need to be uploaded to the Spark session in a separate module, in order to upload as few classes as possible and avoid eventual interaction with the Spark session created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-502) Cleanup Hive dependencies
Marco Gaido created LIVY-502:
Summary: Cleanup Hive dependencies
Key: LIVY-502
URL: https://issues.apache.org/jira/browse/LIVY-502
Project: Livy
Issue Type: Sub-task
Reporter: Marco Gaido
In the starting implementation we are relying/delegating some of the work to
the Hive classes used in the HiveServer2. This helped simplifying the creation
of the first implementation, as it saved to write a lot of code. But this
caused also a dependency on the {{hive-exec}} package, as well as compelled us
to modify a bit some of the existing Hive classes.
The JIRA tracks removing these workarounds by re-implementing the same logic in
Livy to get rid of all Hive dependencies, other than the rpc and service layers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581075#comment-16581075 ] Marco Gaido commented on LIVY-489: -- Sure [~jerryshao], thank you. I am submitting the first PR for 2, 3, 4. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578237#comment-16578237 ] Marco Gaido commented on LIVY-489: -- [~jerryshao] I created 5 subtasks for this. Hope they are reasonable to you. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-495) Add basic UI for thriftserver
Marco Gaido created LIVY-495: Summary: Add basic UI for thriftserver Key: LIVY-495 URL: https://issues.apache.org/jira/browse/LIVY-495 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido The issue tracks the implementation of a UI showing basic information about the status of the Livy thriftserver. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-494) Add thriftserver to Livy server
Marco Gaido created LIVY-494: Summary: Add thriftserver to Livy server Key: LIVY-494 URL: https://issues.apache.org/jira/browse/LIVY-494 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido Including the thriftserver in the Livy server. This means starting the Thriftserver at Livy server startup and adding the needed script in order to interact with it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LIVY-493) Add UTs to the thriftserver module
[ https://issues.apache.org/jira/browse/LIVY-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated LIVY-493: - Description: Tracks the implementation and addition of UT for the new Livy thriftserver. > Add UTs to the thriftserver module > -- > > Key: LIVY-493 > URL: https://issues.apache.org/jira/browse/LIVY-493 > Project: Livy > Issue Type: Sub-task >Reporter: Marco Gaido >Priority: Major > > Tracks the implementation and addition of UT for the new Livy thriftserver. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-493) Add UTs to the thriftserver module
Marco Gaido created LIVY-493: Summary: Add UTs to the thriftserver module Key: LIVY-493 URL: https://issues.apache.org/jira/browse/LIVY-493 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-492) Base implementation Livy thriftserver
Marco Gaido created LIVY-492: Summary: Base implementation Livy thriftserver Key: LIVY-492 URL: https://issues.apache.org/jira/browse/LIVY-492 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido The issue tracks the lading of the initial implementation of the Livy thriftserver -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (LIVY-490) Add thriftserver module
[ https://issues.apache.org/jira/browse/LIVY-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido closed LIVY-490. Resolution: Duplicate > Add thriftserver module > --- > > Key: LIVY-490 > URL: https://issues.apache.org/jira/browse/LIVY-490 > Project: Livy > Issue Type: Sub-task >Reporter: Marco Gaido >Priority: Major > > Add a new module for the Thriftserver implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-491) Add thriftserver module
Marco Gaido created LIVY-491: Summary: Add thriftserver module Key: LIVY-491 URL: https://issues.apache.org/jira/browse/LIVY-491 Project: Livy Issue Type: Sub-task Components: Server Affects Versions: 0.6.0 Reporter: Marco Gaido Add a new module for the implementation of the Livy thriftserver. This includes adding the base thriftserver implementation from Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-490) Add thriftserver module
Marco Gaido created LIVY-490: Summary: Add thriftserver module Key: LIVY-490 URL: https://issues.apache.org/jira/browse/LIVY-490 Project: Livy Issue Type: Sub-task Reporter: Marco Gaido Add a new module for the Thriftserver implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578226#comment-16578226 ] Marco Gaido commented on LIVY-489: -- Sure [~jerryshao], the branch is https://github.com/mgaido91/incubator-livy/tree/livy_thrift, and the diff is https://github.com/apache/incubator-livy/compare/master...mgaido91:livy_thrift. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577995#comment-16577995 ] Marco Gaido commented on LIVY-489: -- Hi [~jerryshao]. Thanks for your comment. Unfortunately I am not sure how to spit the implementation as most of the code is required for it to work. s of now, the only two tasks I have been able to split it into are: - Initial Thriftserver implementation; - Adding a Thriftserver UI. I will keep on thinking on this anyway. Any suggestion is welcomed. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated LIVY-489: - Description: Many users and BI tools use JDBC connections in order to retrieve data. As Livy exposes only a REST API, this is a limitation in its adoption. Hence, adding a JDBC endpoint may be a very useful feature, which could also make Livy a more attractive solution for end user to adopt. Moreover, currently, Spark exposes a JDBC interface, but this has many limitations, including that all the queries are submitted to the same application, therefore there is no isolation/security, which can be offered by Livy, making a Livy JDBC API a better solution for companies/users who want to use Spark in order to run they queries through JDBC. In order to make the transition from existing solutions to the new JDBC server seamless, the proposal is to use the Hive thrift-server and extend it as it was done by the STS. [Here, you can find the design doc.|https://drive.google.com/file/d/10r8aF1xmL2MTtuREawGcrJobMf5Abtts/view?usp=sharing] was: Many users and BI tools use JDBC connections in order to retrieve data. As Livy exposes only a REST API, this is a limitation in its adoption. Hence, adding a JDBC endpoint may be a very useful feature, which could also make Livy a more attractive solution for end user to adopt. Moreover, currently, Spark exposes a JDBC interface, but this has many limitations, including that all the queries are submitted to the same application, therefore there is no isolation/security, which can be offered by Livy, making a Livy JDBC API a better solution for companies/users who want to use Spark in order to run they queries through JDBC. In order to make the transition from existing solutions to the new JDBC server seamless, the proposal is to use the Hive thrift-server and extend it as it was done by the STS. [Here, you can find the design doc.|https://docs.google.com/a/hortonworks.com/document/d/e/2PACX-1vS-ffJwXJ5nZluV-81AJ4WvS3SFX_KcZ0Djz9QGeEtLullYdLHT8dJvuwPpLBT2s3EU4CO6ij14wVcv/pub] > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://drive.google.com/file/d/10r8aF1xmL2MTtuREawGcrJobMf5Abtts/view?usp=sharing] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LIVY-489) Expose a JDBC endpoint for Livy
Marco Gaido created LIVY-489: Summary: Expose a JDBC endpoint for Livy Key: LIVY-489 URL: https://issues.apache.org/jira/browse/LIVY-489 Project: Livy Issue Type: New Feature Components: API, Server Affects Versions: 0.6.0 Reporter: Marco Gaido Many users and BI tools use JDBC connections in order to retrieve data. As Livy exposes only a REST API, this is a limitation in its adoption. Hence, adding a JDBC endpoint may be a very useful feature, which could also make Livy a more attractive solution for end user to adopt. Moreover, currently, Spark exposes a JDBC interface, but this has many limitations, including that all the queries are submitted to the same application, therefore there is no isolation/security, which can be offered by Livy, making a Livy JDBC API a better solution for companies/users who want to use Spark in order to run they queries through JDBC. In order to make the transition from existing solutions to the new JDBC server seamless, the proposal is to use the Hive thrift-server and extend it as it was done by the STS. [Here, you can find the design doc.|https://docs.google.com/a/hortonworks.com/document/d/e/2PACX-1vS-ffJwXJ5nZluV-81AJ4WvS3SFX_KcZ0Djz9QGeEtLullYdLHT8dJvuwPpLBT2s3EU4CO6ij14wVcv/pub] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
