[jira] [Commented] (LIVY-583) python build needs configparser

2019-07-12 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883563#comment-16883563
 ] 

Yiheng Wang commented on LIVY-583:
--

It is configured in the setup.py.

[https://github.com/apache/incubator-livy/blob/master/python-api/setup.py#L32]

> python build needs configparser
> ---
>
> Key: LIVY-583
> URL: https://issues.apache.org/jira/browse/LIVY-583
> Project: Livy
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Felix Cheung
>Priority: Major
>
> pip install configparser  just to build. it won't build until I manually pip 
> install.
>  
> (run into this in my 2nd environment)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Issue Comment Deleted] (LIVY-583) python build needs configparser

2019-07-12 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-583:
-
Comment: was deleted

(was: It is configured in the setup.py.

[https://github.com/apache/incubator-livy/blob/master/python-api/setup.py#L32])

> python build needs configparser
> ---
>
> Key: LIVY-583
> URL: https://issues.apache.org/jira/browse/LIVY-583
> Project: Livy
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Felix Cheung
>Priority: Major
>
> pip install configparser  just to build. it won't build until I manually pip 
> install.
>  
> (run into this in my 2nd environment)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-583) python build needs configparser

2019-07-12 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883769#comment-16883769
 ] 

Yiheng Wang commented on LIVY-583:
--

configparser is already in the setup.py, 
[https://github.com/apache/incubator-livy/blob/master/python-api/setup.py#L32]

> python build needs configparser
> ---
>
> Key: LIVY-583
> URL: https://issues.apache.org/jira/browse/LIVY-583
> Project: Livy
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Felix Cheung
>Priority: Major
>
> pip install configparser  just to build. it won't build until I manually pip 
> install.
>  
> (run into this in my 2nd environment)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-582) python test_create_new_session_without_default_config test fails consistently

2019-07-12 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883767#comment-16883767
 ] 

Yiheng Wang commented on LIVY-582:
--

Just change the upper case characters in your hostname into lower case and the 
issue will go.:P

The root cause is the python Request lib will change the request url to lower 
case, while the mock lib is case sensitive...

> python test_create_new_session_without_default_config test fails consistently
> -
>
> Key: LIVY-582
> URL: https://issues.apache.org/jira/browse/LIVY-582
> Project: Livy
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Felix Cheung
>Priority: Major
>
> {code:java}
> test_create_new_session_without_default_config 
> def test_create_new_session_without_default_config():
> > mock_and_validate_create_new_session(False)
> src/test/python/livy-tests/client_test.py:105:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> _
> :3: in wrapper
> ???
> src/test/python/livy-tests/client_test.py:48: in 
> mock_and_validate_create_new_session
> load_defaults=defaults)
> src/main/python/livy/client.py:88: in __init__
> session_conf_dict).json()['id']
> src/main/python/livy/client.py:388: in _create_new_session
> headers=self._conn._JSON_HEADERS, data=data)
> src/main/python/livy/client.py:500: in send_request
> json=data, auth=self._spnego_auth())
> .eggs/requests-2.21.0-py2.7.egg/requests/api.py:60: in request
> return session.request(method=method, url=url, **kwargs)
> .eggs/requests-2.21.0-py2.7.egg/requests/sessions.py:533: in request
> resp = self.send(prep, **send_kwargs)
> .eggs/requests-2.21.0-py2.7.egg/requests/sessions.py:646: in send
> r = adapter.send(request, **kwargs)
> .eggs/responses-0.10.6-py2.7.egg/responses.py:626: in unbound_on_send
> return self._on_request(adapter, request, *a, **kwargs)
> self = 
> adapter = 
> request = 
> kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': 
> 10, ...}
> match = None, resp_callback = None
> error_msg = "Connection refused by Responses: POST 
> http://machine:8998/sessions/ doesn't match Responses Mock"
> response = ConnectionError(u"Connection refused by Responses: POST 
> http://machine:8998/sessions/doesn't match Responses Mock",)
> {code}
> Not sure why. this fails 100% and I don't see anything listening to this 
> port. Need some help to troubleshoot this.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-637) get NullPointerException when create database using thriftserver

2019-08-14 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907012#comment-16907012
 ] 

Yiheng Wang commented on LIVY-637:
--

Spark beeline uses an older hive-jdbc-client version, in which it will not 
check if the column is null in the RowSet returned by the thrift server.

 

See here:

[https://github.com/apache/hive/blob/release-1.2.1/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L50]

 

In the latest hive code, this has been fixed

[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L82]

> get NullPointerException when create database using thriftserver
> 
>
> Key: LIVY-637
> URL: https://issues.apache.org/jira/browse/LIVY-637
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: mingchao zhao
>Priority: Major
> Attachments: create.png, drop.png, use.png
>
>
> When I connected thriftserver with spark beeline. NullPointerException occurs 
> when  execute the following SQL. This exception does not affect the final 
> execution result.
> create database test;
> use test;
> drop database test;
> 0: jdbc:hive2://localhost:10090> create database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  0: jdbc:hive2://localhost:10090> use test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
> 0: jdbc:hive2://localhost:10090> drop database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Comment Edited] (LIVY-637) get NullPointerException when create database using thriftserver

2019-08-14 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907012#comment-16907012
 ] 

Yiheng Wang edited comment on LIVY-637 at 8/14/19 8:25 AM:
---

Spark beeline uses an old hive-jdbc-client version, in which it will not check 
if the column is null in the RowSet returned by the thrift server.

 

See here:

[https://github.com/apache/hive/blob/release-1.2.1/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L50]

 

In the latest hive code, this has been fixed

[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L82]


was (Author: yihengw):
Spark beeline uses an older hive-jdbc-client version, in which it will not 
check if the column is null in the RowSet returned by the thrift server.

 

See here:

[https://github.com/apache/hive/blob/release-1.2.1/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L50]

 

In the latest hive code, this has been fixed

[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L82]

> get NullPointerException when create database using thriftserver
> 
>
> Key: LIVY-637
> URL: https://issues.apache.org/jira/browse/LIVY-637
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: mingchao zhao
>Priority: Major
> Attachments: create.png, drop.png, use.png
>
>
> When I connected thriftserver with spark beeline. NullPointerException occurs 
> when  execute the following SQL. This exception does not affect the final 
> execution result.
> create database test;
> use test;
> drop database test;
> 0: jdbc:hive2://localhost:10090> create database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  0: jdbc:hive2://localhost:10090> use test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
> 0: jdbc:hive2://localhost:10090> drop database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-573) Add tests for operation logs retrieval

2019-08-14 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907199#comment-16907199
 ] 

Yiheng Wang commented on LIVY-573:
--

Have created a patch. [~mgaido] can you please help review it? Thanks

> Add tests for operation logs retrieval
> --
>
> Key: LIVY-573
> URL: https://issues.apache.org/jira/browse/LIVY-573
> Project: Livy
>  Issue Type: Improvement
>  Components: Tests, Thriftserver
>Reporter: Marco Gaido
>Priority: Trivial
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Our current tests do not cover the retrieval of operation logs. We should try 
> and add coverage for it if possible.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-633) session should not be gc-ed for long running queries

2019-08-15 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907993#comment-16907993
 ] 

Yiheng Wang commented on LIVY-633:
--

Is this related to this JIRA? 
[https://issues.apache.org/jira/projects/LIVY/issues/LIVY-547?filter=allissues]

> session should not be gc-ed for long running queries
> 
>
> Key: LIVY-633
> URL: https://issues.apache.org/jira/browse/LIVY-633
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Liju
>Priority: Major
>
> If you have set a relatively small session timeout eg 15 mins and query 
> execution is taking > 15 mins , the session gets gc-ed , which is incorrect 
> wrt user experience as the user was still active on session and waiting for 
> result



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-638) get sql.AnalysisException when create table using thriftserver

2019-08-14 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907028#comment-16907028
 ] 

Yiheng Wang commented on LIVY-638:
--

It's weird that even I enable hive support in spark configuration, livy thrift 
server will still throw this exception... 

> get sql.AnalysisException when create table using thriftserver
> --
>
> Key: LIVY-638
> URL: https://issues.apache.org/jira/browse/LIVY-638
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: mingchao zhao
>Priority: Major
> Attachments: create table.png
>
>
> org.apache.spark.sql.AnalysisExceptionoccurs when I use thriftserver to 
> execute the following SQL. When I do not use hive as metastore, thriftserver 
> does not support create table ?
> 0: jdbc:hive2://localhost:10090> CREATE TABLE test(key INT, val STRING);
>  Error: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> org.apache.spark.sql.AnalysisException: Hive support is required to CREATE 
> Hive TABLE (AS SELECT);;
>  'CreateTable `test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, 
> ErrorIfExists
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:392)
>  
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:390)
>  org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:117)
>  
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:390)
>  
> org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:388)
>  
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
>  
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
>  
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>  scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>  
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:386)
>  
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
>  
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
>  
> org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
>  
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
>  
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
>  
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
>  
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
>  
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
>  org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
>  org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
>  org.apache.livy.thriftserver.session.SqlJob.executeSql(SqlJob.java:74)
>  org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:64)
>  org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:35)
>  org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
>  org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  java.lang.Thread.run(Thread.java:748) (state=,code=0)
>  0: jdbc:hive2://localhost:10090>
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-574) Add tests for metadata operations

2019-08-12 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905849#comment-16905849
 ] 

Yiheng Wang commented on LIVY-574:
--

I found some errors relate with existing meta operation when I test it with 
squirrel-SQL. I would like to take this one and fix these issues.

> Add tests for metadata operations
> -
>
> Key: LIVY-574
> URL: https://issues.apache.org/jira/browse/LIVY-574
> Project: Livy
>  Issue Type: Improvement
>  Components: Tests, Thriftserver
>Reporter: Marco Gaido
>Priority: Major
>
> We do not have tests for metadata operations. We should add them at least for 
> the ones which we have currently implemented.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (LIVY-635) Travis failed to build

2019-08-13 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-635:
-
Component/s: Tests

> Travis failed to build
> --
>
> Key: LIVY-635
> URL: https://issues.apache.org/jira/browse/LIVY-635
> Project: Livy
>  Issue Type: Bug
>  Components: Tests, Thriftserver
>Affects Versions: 0.6.0
>Reporter: jiewang
>Priority: Major
>
> [ERROR] Failed to execute goal on project livy-thriftserver: Could not 
> resolve dependencies for project 
> org.apache.livy:livy-thriftserver:jar:0.7.0-incubating-SNAPSHOT: Failed to 
> collect dependencies at org.apache.hive:hive-jdbc:jar:3.0.0 -> 
> org.apache.hive:hive-service:jar:3.0.0 -> 
> org.apache.hive:hive-llap-server:jar:3.0.0 -> 
> org.apache.hbase:hbase-server:jar:2.0.0-alpha4 -> 
> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 -> 
> org.glassfish:javax.el:jar:3.0.1-b08-SNAPSHOT: Failed to read artifact 
> descriptor for org.glassfish:javax.el:jar:3.0.1-b08-SNAPSHOT: Could not 
> transfer artifact org.glassfish:javax.el:pom:3.0.1-b08-SNAPSHOT from/to 
> apache-snapshots (https://repository.apache.org/snapshots/): Connect to 
> repository.apache.org:443 [repository.apache.org/207.244.88.140] failed: 
> Connection timed out (Connection timed out) -> [Help 1]
> 2258org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal on project livy-thriftserver: Could not resolve dependencies for project 
> org.apache.livy:livy-thriftserver:jar:0.7.0-incubating-SNAPSHOT: Failed to 
> collect dependencies at org.apache.hive:hive-jdbc:jar:3.0.0 -> 
> org.apache.hive:hive-service:jar:3.0.0 -> 
> org.apache.hive:hive-llap-server:jar:3.0.0 -> 
> org.apache.hbase:hbase-server:jar:2.0.0-alpha4 -> 
> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 -> 
> org.glassfish:javax.el:jar:3.0.1-b08-SNAPSHOT
> 2259 at 
> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies
>  (LifecycleDependencyResolver.java:249)
> 2260 at 
> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies
>  (LifecycleDependencyResolver.java:145)
> 2261 at 
> org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved
>  (MojoExecutor.java:246)
> 2262 at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:200)
> 2263 at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> 2264 at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> 2265 at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> 2266 at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> 2267 at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> 2268 at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> 2269 at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> 2270 at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> 2271 at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> 2272 at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> 2273 at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> 2274 at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> 2275 at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> 2276 at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> 2277 at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> 2278 at java.lang.reflect.Method.invoke (Method.java:498)
> 2279 at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> 2280 at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> 2281 at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> 2282 at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-178) PySpark sessions crash on invalid string

2019-08-19 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910924#comment-16910924
 ] 

Yiheng Wang commented on LIVY-178:
--

This was reported in 0.2. Maybe it has been fixed in the latest version.

> PySpark sessions crash on invalid string
> 
>
> Key: LIVY-178
> URL: https://issues.apache.org/jira/browse/LIVY-178
> Project: Livy
>  Issue Type: Bug
>  Components: Interpreter
>Affects Versions: 0.2
> Environment: {code}
> print "\xHH"
> {code}
> will crash the session
>Reporter: Alex Man
>Assignee: Alex Man
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (LIVY-515) Livy session is idle even if spark context is unavailable

2019-08-19 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910923#comment-16910923
 ] 

Yiheng Wang commented on LIVY-515:
--

This is reported in 0.4.0. Maybe this issue has been fixed in latest version.

> Livy session is idle even if spark context is unavailable
> -
>
> Key: LIVY-515
> URL: https://issues.apache.org/jira/browse/LIVY-515
> Project: Livy
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.4.0
>Reporter: Raghavendra
>Priority: Major
> Attachments: AfterKillingLivy.png, BeforeKillingLivy.png, 
> command.history, livy-log, livyUI.png
>
>
> When using livy in the session mode, Livy is not able to figure out if the 
> application master is available.
> Say suppose livy was in the idle state, now for some reason the application 
> master died. Ideally, even the session associated with this application 
> master should either be terminated or should have handled this situation. 
> Instead, the session continues to remain in the Idle state. This is wrong.
>  
> Steps to reproduce.
>  # Bring up livy in session mode(I am using 5 sessions).
>  # Kill the application master using "yarn application -kill "
>  # Now list the application in yarn "yarn application  -list" you will see 4 
> application master(I had 5, Killed 1 so 4)
>  # Now reload livy UI. You will still see 5 sessions in idle.
> Attached the session logs whose application master was killed.
> Attached are some images depicting the issue



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Comment Edited] (LIVY-515) Livy session is idle even if spark context is unavailable

2019-08-19 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910923#comment-16910923
 ] 

Yiheng Wang edited comment on LIVY-515 at 8/20/19 2:14 AM:
---

This was reported in 0.4.0. Maybe this issue has been fixed in the latest 
version.


was (Author: yihengw):
This is reported in 0.4.0. Maybe this issue has been fixed in latest version.

> Livy session is idle even if spark context is unavailable
> -
>
> Key: LIVY-515
> URL: https://issues.apache.org/jira/browse/LIVY-515
> Project: Livy
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.4.0
>Reporter: Raghavendra
>Priority: Major
> Attachments: AfterKillingLivy.png, BeforeKillingLivy.png, 
> command.history, livy-log, livyUI.png
>
>
> When using livy in the session mode, Livy is not able to figure out if the 
> application master is available.
> Say suppose livy was in the idle state, now for some reason the application 
> master died. Ideally, even the session associated with this application 
> master should either be terminated or should have handled this situation. 
> Instead, the session continues to remain in the Idle state. This is wrong.
>  
> Steps to reproduce.
>  # Bring up livy in session mode(I am using 5 sessions).
>  # Kill the application master using "yarn application -kill "
>  # Now list the application in yarn "yarn application  -list" you will see 4 
> application master(I had 5, Killed 1 so 4)
>  # Now reload livy UI. You will still see 5 sessions in idle.
> Attached the session logs whose application master was killed.
> Attached are some images depicting the issue



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Created] (LIVY-644) Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report

2019-08-22 Thread Yiheng Wang (Jira)
Yiheng Wang created LIVY-644:


 Summary: Flaky test: Failed to execute goal 
org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
project livy-coverage-report
 Key: LIVY-644
 URL: https://issues.apache.org/jira/browse/LIVY-644
 Project: Livy
  Issue Type: Improvement
  Components: Tests
Reporter: Yiheng Wang


Recently a lot of Travis job failed when generating coverage report:
[https://travis-ci.org/apache/incubator-livy/jobs/575142847]
[https://travis-ci.org/apache/incubator-livy/jobs/561700903]
[https://travis-ci.org/apache/incubator-livy/jobs/508574433]
[https://travis-ci.org/apache/incubator-livy/jobs/508574435]
[https://travis-ci.org/apache/incubator-livy/jobs/508066760]
[https://travis-ci.org/apache/incubator-livy/jobs/507989073]
[https://travis-ci.org/apache/incubator-livy/jobs/574702251]
[https://travis-ci.org/apache/incubator-livy/jobs/574686891]
[https://travis-ci.org/apache/incubator-livy/jobs/574363881]
[https://travis-ci.org/apache/incubator-livy/jobs/574215174]
[https://travis-ci.org/apache/incubator-livy/jobs/573689926]
 

Here is the error stack:

 
[ERROR] Failed to execute goal 
org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
project livy-coverage-report: An error has occurred in JaCoCo Aggregate report 
generation. Error while creating report: null: EOFException -> [Help 1]
2988org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
project livy-coverage-report: An error has occurred in JaCoCo Aggregate report 
generation.
2989at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:213)
2990at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
2991at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
2992at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
2993at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
2994at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:51)
2995at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
2996at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
2997at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
2998at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
2999at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
3000at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
3001at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
3002at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
3003at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
3004at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
3005at java.lang.reflect.Method.invoke (Method.java:498)
3006at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
3007at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
3008at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
3009at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
3010Caused by: org.apache.maven.plugin.MojoExecutionException: An error has 
occurred in JaCoCo Aggregate report generation.
3011at org.jacoco.maven.AbstractReportMojo.execute (AbstractReportMojo.java:167)
3012at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:134)
3013at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:208)
3014at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
3015at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
3016at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
3017at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
3018at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:51)
3019at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
3020at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
3021at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
3022at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
3023at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
3024at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
3025at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
3026at sun.reflect.NativeMethodAccessorImpl.invoke0 

[jira] [Updated] (LIVY-644) Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report

2019-08-22 Thread Yiheng Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/LIVY-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-644:
-
Issue Type: Bug  (was: Improvement)

> Flaky test: Failed to execute goal 
> org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
> project livy-coverage-report
> 
>
> Key: LIVY-644
> URL: https://issues.apache.org/jira/browse/LIVY-644
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Reporter: Yiheng Wang
>Priority: Minor
>
> Recently a lot of Travis job failed when generating coverage report:
> [https://travis-ci.org/apache/incubator-livy/jobs/575142847]
> [https://travis-ci.org/apache/incubator-livy/jobs/561700903]
> [https://travis-ci.org/apache/incubator-livy/jobs/508574433]
> [https://travis-ci.org/apache/incubator-livy/jobs/508574435]
> [https://travis-ci.org/apache/incubator-livy/jobs/508066760]
> [https://travis-ci.org/apache/incubator-livy/jobs/507989073]
> [https://travis-ci.org/apache/incubator-livy/jobs/574702251]
> [https://travis-ci.org/apache/incubator-livy/jobs/574686891]
> [https://travis-ci.org/apache/incubator-livy/jobs/574363881]
> [https://travis-ci.org/apache/incubator-livy/jobs/574215174]
> [https://travis-ci.org/apache/incubator-livy/jobs/573689926]
>  
> Here is the error stack:
>  
> [ERROR] Failed to execute goal 
> org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
> project livy-coverage-report: An error has occurred in JaCoCo Aggregate 
> report generation. Error while creating report: null: EOFException -> [Help 1]
> 2988org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
> project livy-coverage-report: An error has occurred in JaCoCo Aggregate 
> report generation.
> 2989at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:213)
> 2990at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> 2991at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> 2992at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> 2993at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> 2994at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> 2995at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> 2996at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> 2997at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> 2998at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> 2999at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> 3000at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> 3001at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> 3002at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> 3003at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> 3004at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> 3005at java.lang.reflect.Method.invoke (Method.java:498)
> 3006at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> 3007at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> 3008at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> 3009at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> 3010Caused by: org.apache.maven.plugin.MojoExecutionException: An error has 
> occurred in JaCoCo Aggregate report generation.
> 3011at org.jacoco.maven.AbstractReportMojo.execute 
> (AbstractReportMojo.java:167)
> 3012at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> 3013at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> 3014at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> 3015at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> 3016at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> 3017at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> 3018at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> 3019at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 

[jira] [Commented] (LIVY-633) session should not be gc-ed for long running queries

2019-09-03 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921215#comment-16921215
 ] 

Yiheng Wang commented on LIVY-633:
--

Yes, it's a different problem. It should be a bug in Livy. I'm working on a 
patch to fix it.

> session should not be gc-ed for long running queries
> 
>
> Key: LIVY-633
> URL: https://issues.apache.org/jira/browse/LIVY-633
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Liju
>Priority: Major
>
> If you have set a relatively small session timeout eg 15 mins and query 
> execution is taking > 15 mins , the session gets gc-ed , which is incorrect 
> wrt user experience as the user was still active on session and waiting for 
> result



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Comment Edited] (LIVY-664) Spark application still running when Livy session creating was rejected

2019-09-12 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928486#comment-16928486
 ] 

Yiheng Wang edited comment on LIVY-664 at 9/12/19 12:18 PM:


In the code, it will stop the duplicated session when finding there're 
duplicated names. In my test, I find the stop session may fail. I guess it may 
due to some race condition. I'm looking into it.

Another suggestion I think it's great. We should check the name duplication 
before submitting the application. I'll raise a patch to fix that.


was (Author: yihengw):
In the code, it will stop the duplicated session when finding there're 
duplicated names. In my test, I find the stop session may fail. I guess it may 
due to some race condition. I'm looking into it.

Another suggestion I think it good. We should check the name duplication before 
submitting the application. I'll raise a patch to fix that.

> Spark application still running when Livy session creating was rejected 
> 
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png, 
> image-2019-09-08-20-39-18-569.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
>   2.1 Second session creating will be rejected since duplicated session name 
> is not allowed.
>   2.2 Spark application will be submitted but Livy session won't be created
>  
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should 
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected

2019-09-12 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928486#comment-16928486
 ] 

Yiheng Wang commented on LIVY-664:
--

In the code, it will stop the duplicated session when finding there're 
duplicated names. In my test, I find the stop session may fail. I guess it may 
due to some race condition. I'm looking into it.

Another suggestion I think it good. We should check the name duplication before 
submitting the application. I'll raise a patch to fix that.

> Spark application still running when Livy session creating was rejected 
> 
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png, 
> image-2019-09-08-20-39-18-569.png
>
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
>   2.1 Second session creating will be rejected since duplicated session name 
> is not allowed.
>   2.2 Spark application will be submitted but Livy session won't be created
>  
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should 
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected

2019-09-17 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931300#comment-16931300
 ] 

Yiheng Wang commented on LIVY-664:
--

session stop code: 
[https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L103]

> Spark application still running when Livy session creating was rejected 
> 
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png, 
> image-2019-09-08-20-39-18-569.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
>   2.1 Second session creating will be rejected since duplicated session name 
> is not allowed.
>   2.2 Spark application will be submitted but Livy session won't be created
>  
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should 
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected

2019-09-17 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931261#comment-16931261
 ] 

Yiheng Wang commented on LIVY-664:
--

Oh, I check the wrong branch... Sorry for the confusion.

 

 

> Spark application still running when Livy session creating was rejected 
> 
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png, 
> image-2019-09-08-20-39-18-569.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
>   2.1 Second session creating will be rejected since duplicated session name 
> is not allowed.
>   2.2 Spark application will be submitted but Livy session won't be created
>  
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should 
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Issue Comment Deleted] (LIVY-664) Spark application still running when Livy session creating was rejected

2019-09-17 Thread Yiheng Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-664:
-
Comment: was deleted

(was: Oh, I check the wrong branch... Sorry for the confusion.

 

 )

> Spark application still running when Livy session creating was rejected 
> 
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png, 
> image-2019-09-08-20-39-18-569.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
>   2.1 Second session creating will be rejected since duplicated session name 
> is not allowed.
>   2.2 Spark application will be submitted but Livy session won't be created
>  
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should 
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (LIVY-660) How can we use YARN and all the nodes in our cluster when submiting a pySpark job

2019-09-05 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923112#comment-16923112
 ] 

Yiheng Wang commented on LIVY-660:
--

I think it's better to post such question in livy user email group.

 

You just post your livy-client.conf. Do you modify your livy.conf?

> How can we use YARN and all the nodes in our cluster when submiting a pySpark 
> job
> -
>
> Key: LIVY-660
> URL: https://issues.apache.org/jira/browse/LIVY-660
> Project: Livy
>  Issue Type: Question
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Sebastian Rama
>Priority: Minor
>
> How can we use YARN and all the nodes in our cluster when submiting a pySpark 
> job?
> We have edited all the required .conf files but nothing happens. =(
>  
>  
> [root@cdh-node06 conf]# cat livy-client.conf
> #
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements.  See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License.  You may obtain a copy of the License at
> #
> #    http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #
> # Use this keystore for the SSL certificate and key.
> # livy.keystore =
> dew0wf-e
> # Specify the keystore password.
> # livy.keystore.password =
> #
> welfka
> # Specify the key password.
> # livy.key-password =
>  
> # Hadoop Credential Provider Path to get "livy.keystore.password" and 
> "livy.key-password".
> # Credential Provider can be created using command as follow:
> # hadoop credential create "livy.keystore.password" -value "secret" -provider 
> jceks://hdfs/path/to/livy.jceks
> # livy.hadoop.security.credential.provider.path =
>  
> # What host address to start the server on. By default, Livy will bind to all 
> network interfaces.
> # livy.server.host = 0.0.0.0
>  
> # What port to start the server on.
> # livy.server.port = 8998
>  
> # What base path ui should work on. By default UI is mounted on "/".
> # E.g.: livy.ui.basePath = /my_livy - result in mounting UI on /my_livy/
> # livy.ui.basePath = ""
>  
> # What spark master Livy sessions should use.
> livy.spark.master = yarn
>  
> # What spark deploy mode Livy sessions should use.
> livy.spark.deploy-mode = cluster
>  
> # Configure Livy server http request and response header size.
> # livy.server.request-header.size = 131072
> # livy.server.response-header.size = 131072
>  
> # Enabled to check whether timeout Livy sessions should be stopped.
> # livy.server.session.timeout-check = true
>  
> # Time in milliseconds on how long Livy will wait before timing out an idle 
> session.
> # livy.server.session.timeout = 1h
> #
> # How long a finished session state should be kept in LivyServer for query.
> # livy.server.session.state-retain.sec = 600s
>  
> # If livy should impersonate the requesting users when creating a new session.
> # livy.impersonation.enabled = true
>  
> # Logs size livy can cache for each session/batch. 0 means don't cache the 
> logs.
> # livy.cache-log.size = 200
>  
> # Comma-separated list of Livy RSC jars. By default Livy will upload jars 
> from its installation
> # directory every time a session is started. By caching these files in HDFS, 
> for example, startup
> # time of sessions on YARN can be reduced.
> # livy.rsc.jars =
>  
> # Comma-separated list of Livy REPL jars. By default Livy will upload jars 
> from its installation
> # directory every time a session is started. By caching these files in HDFS, 
> for example, startup
> # time of sessions on YARN can be reduced. Please list all the repl 
> dependencies including
> # Scala version-specific livy-repl jars, Livy will automatically pick the 
> right dependencies
> # during session creation.
> # livy.repl.jars =
>  
> # Location of PySpark archives. By default Livy will upload the file from 
> SPARK_HOME, but
> # by caching the file in HDFS, startup time of PySpark sessions on YARN can 
> be reduced.
> # livy.pyspark.archives =
>  
> # Location of the SparkR package. By default Livy will upload the file from 
> SPARK_HOME, but
> # by caching the file in HDFS, startup time of R sessions on YARN can be 
> reduced.
> # livy.sparkr.package =
>  
> # List of local directories from where files are allowed to be added to user 
> 

[jira] [Commented] (LIVY-684) Livy server support zookeeper service discovery

2019-09-19 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933110#comment-16933110
 ] 

Yiheng Wang commented on LIVY-684:
--

This is duplicate with https://issues.apache.org/jira/browse/LIVY-616

> Livy server support zookeeper service discovery
> ---
>
> Key: LIVY-684
> URL: https://issues.apache.org/jira/browse/LIVY-684
> Project: Livy
>  Issue Type: New Feature
>Reporter: Zhefeng Wang
>Priority: Minor
>
> Livy server hasn't support service discovery, which is widely used in highly 
> available. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (LIVY-615) livy.ui.basePath does not seem to work correctly

2019-07-30 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896746#comment-16896746
 ] 

Yiheng Wang commented on LIVY-615:
--

Can you raise a PR for your patch?

> livy.ui.basePath does not seem to work correctly
> 
>
> Key: LIVY-615
> URL: https://issues.apache.org/jira/browse/LIVY-615
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Ferdinand de Antoni
>Priority: Major
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> When setting the property {{livy.ui.basePath}},  a HTTP error 404 is returned.
> To resolve this problem, the context path in {{WebServer.scala}} should be 
> set as well, e.g.:
> {code:java}
> context.setContextPath(livyConf.get(LivyConf.SERVER_BASE_PATH)){code}
> Adding this seems to resolve this problem. Note that this of course also 
> changes the context path of the API, not just the UI, but I presumed that was 
> also the intention of the {{livy.ui.basePath}} property.
> Below patch of suggested changes:
> {noformat}
> Index: server/src/main/scala/org/apache/livy/server/WebServer.scala
> IDEA additional info:
> Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
> <+>UTF-8
> ===
> --- server/src/main/scala/org/apache/livy/server/WebServer.scala   (revision 
> 92062e1659db2af85711b1f35c50ff4050fec675)
> +++ server/src/main/scala/org/apache/livy/server/WebServer.scala   (revision 
> bdfb75d08bf34633ff23d7e4db380aca1fdf4d8e)
> @@ -81,7 +81,7 @@
>val context = new ServletContextHandler()
> -  context.setContextPath("/")
> +  context.setContextPath(livyConf.get(LivyConf.SERVER_BASE_PATH))
>context.addServlet(classOf[DefaultServlet], "/")
>val handlers = new HandlerCollection
> @@ -114,7 +114,7 @@
>  }
>  port = connector.getLocalPort
> -info("Starting server on %s://%s:%d" format (protocol, host, port))
> +info("Starting server on %s://%s:%d/%s" format (protocol, host, port, 
> livyConf.get(LivyConf.SERVER_BASE_PATH)))
>}
>def join(): Unit = {{noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-626) Implement GetPrimaryKeys metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-626:


 Summary: Implement GetPrimaryKeys metadata operation
 Key: LIVY-626
 URL: https://issues.apache.org/jira/browse/LIVY-626
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetPrimaryKeys metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-628) Implement GetDelegationToken metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-628:


 Summary: Implement GetDelegationToken metadata operation
 Key: LIVY-628
 URL: https://issues.apache.org/jira/browse/LIVY-628
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetDelegationToken metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-627) Implement GetCrossReference metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-627:


 Summary: Implement GetCrossReference metadata operation
 Key: LIVY-627
 URL: https://issues.apache.org/jira/browse/LIVY-627
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetCrossReference metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-630) Implement RenewDelegationToken metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-630:


 Summary: Implement RenewDelegationToken metadata operation
 Key: LIVY-630
 URL: https://issues.apache.org/jira/browse/LIVY-630
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support RenewDelegationToken metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-629) Implement CancelDelegationToken metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-629:


 Summary: Implement CancelDelegationToken metadata operation
 Key: LIVY-629
 URL: https://issues.apache.org/jira/browse/LIVY-629
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support CancelDelegationToken metadata operation in Livy thrift 
server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-624) Implement GetColumns metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-624:


 Summary: Implement GetColumns metadata operation
 Key: LIVY-624
 URL: https://issues.apache.org/jira/browse/LIVY-624
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetColumns metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-625) Implement GetFunctions metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-625:


 Summary: Implement GetFunctions metadata operation
 Key: LIVY-625
 URL: https://issues.apache.org/jira/browse/LIVY-625
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetFunctions metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-623) Implement GetTables metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-623:


 Summary: Implement GetTables metadata operation
 Key: LIVY-623
 URL: https://issues.apache.org/jira/browse/LIVY-623
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetTables metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-632) Implement SetClientInfo metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-632:


 Summary: Implement SetClientInfo metadata operation
 Key: LIVY-632
 URL: https://issues.apache.org/jira/browse/LIVY-632
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support SetClientInfo metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-631) Implement GetQueryId metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-631:


 Summary: Implement GetQueryId metadata operation
 Key: LIVY-631
 URL: https://issues.apache.org/jira/browse/LIVY-631
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang


We should support GetSchemas metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (LIVY-631) Implement GetQueryId metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-631:
-
Description: We should support GetQueryId metadata operation in Livy thrift 
server.  (was: We should support GetSchemas metadata operation in Livy thrift 
server.)

> Implement GetQueryId metadata operation
> ---
>
> Key: LIVY-631
> URL: https://issues.apache.org/jira/browse/LIVY-631
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Assignee: Yiheng Wang
>Priority: Minor
>
> We should support GetQueryId metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-622) Implement GetSchemas metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)
Yiheng Wang created LIVY-622:


 Summary: Implement GetSchemas metadata operation
 Key: LIVY-622
 URL: https://issues.apache.org/jira/browse/LIVY-622
 Project: Livy
  Issue Type: Sub-task
  Components: Thriftserver
Reporter: Yiheng Wang
Assignee: Yiheng Wang






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (LIVY-622) Implement GetSchemas metadata operation

2019-08-05 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-622:
-
Description: We should support GetSchemas metadata operation in Livy thrift 
server.

> Implement GetSchemas metadata operation
> ---
>
> Key: LIVY-622
> URL: https://issues.apache.org/jira/browse/LIVY-622
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Assignee: Yiheng Wang
>Priority: Minor
>
> We should support GetSchemas metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Assigned] (LIVY-575) Implement missing metadata operations

2019-08-05 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang reassigned LIVY-575:


Assignee: Yiheng Wang

> Implement missing metadata operations
> -
>
> Key: LIVY-575
> URL: https://issues.apache.org/jira/browse/LIVY-575
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Marco Gaido
>Assignee: Yiheng Wang
>Priority: Minor
>
> Many metadata operations (eg. table list retrieval, schema retrieval, ...) 
> are currently not implemented. We should implement them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-624) Implement GetColumns metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900968#comment-16900968
 ] 

Yiheng Wang commented on LIVY-624:
--

Working on it.

> Implement GetColumns metadata operation
> ---
>
> Key: LIVY-624
> URL: https://issues.apache.org/jira/browse/LIVY-624
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetColumns metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-623) Implement GetTables metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900967#comment-16900967
 ] 

Yiheng Wang commented on LIVY-623:
--

Working on it.

> Implement GetTables metadata operation
> --
>
> Key: LIVY-623
> URL: https://issues.apache.org/jira/browse/LIVY-623
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetTables metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-622) Implement GetSchemas metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900966#comment-16900966
 ] 

Yiheng Wang commented on LIVY-622:
--

Working on it.

> Implement GetSchemas metadata operation
> ---
>
> Key: LIVY-622
> URL: https://issues.apache.org/jira/browse/LIVY-622
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetSchemas metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Assigned] (LIVY-630) Implement RenewDelegationToken metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang reassigned LIVY-630:


Assignee: (was: Yiheng Wang)

> Implement RenewDelegationToken metadata operation
> -
>
> Key: LIVY-630
> URL: https://issues.apache.org/jira/browse/LIVY-630
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support RenewDelegationToken metadata operation in Livy thrift 
> server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-08-06 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900963#comment-16900963
 ] 

Yiheng Wang commented on LIVY-575:
--

Got it. Sorry for the inconvenience.

> Implement missing metadata operations
> -
>
> Key: LIVY-575
> URL: https://issues.apache.org/jira/browse/LIVY-575
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Marco Gaido
>Priority: Minor
>
> Many metadata operations (eg. table list retrieval, schema retrieval, ...) 
> are currently not implemented. We should implement them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Assigned] (LIVY-631) Implement GetQueryId metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang reassigned LIVY-631:


Assignee: (was: Yiheng Wang)

> Implement GetQueryId metadata operation
> ---
>
> Key: LIVY-631
> URL: https://issues.apache.org/jira/browse/LIVY-631
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetQueryId metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-625) Implement GetFunctions metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900969#comment-16900969
 ] 

Yiheng Wang commented on LIVY-625:
--

Working on it.

> Implement GetFunctions metadata operation
> -
>
> Key: LIVY-625
> URL: https://issues.apache.org/jira/browse/LIVY-625
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetFunctions metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Assigned] (LIVY-632) Implement SetClientInfo metadata operation

2019-08-06 Thread Yiheng Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/LIVY-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang reassigned LIVY-632:


Assignee: (was: Yiheng Wang)

> Implement SetClientInfo metadata operation
> --
>
> Key: LIVY-632
> URL: https://issues.apache.org/jira/browse/LIVY-632
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support SetClientInfo metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-623) Implement GetTables metadata operation

2019-08-08 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902987#comment-16902987
 ] 

Yiheng Wang commented on LIVY-623:
--

[https://github.com/apache/incubator-livy/pull/194]

> Implement GetTables metadata operation
> --
>
> Key: LIVY-623
> URL: https://issues.apache.org/jira/browse/LIVY-623
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetTables metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-624) Implement GetColumns metadata operation

2019-08-08 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902988#comment-16902988
 ] 

Yiheng Wang commented on LIVY-624:
--

[https://github.com/apache/incubator-livy/pull/194]

> Implement GetColumns metadata operation
> ---
>
> Key: LIVY-624
> URL: https://issues.apache.org/jira/browse/LIVY-624
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetColumns metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-625) Implement GetFunctions metadata operation

2019-08-08 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902989#comment-16902989
 ] 

Yiheng Wang commented on LIVY-625:
--

[https://github.com/apache/incubator-livy/pull/194]

> Implement GetFunctions metadata operation
> -
>
> Key: LIVY-625
> URL: https://issues.apache.org/jira/browse/LIVY-625
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Minor
>
> We should support GetFunctions metadata operation in Livy thrift server.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-07-22 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890653#comment-16890653
 ] 

Yiheng Wang commented on LIVY-575:
--

[~mgaido] Looks like PR-182 is to fix Jira-571 instead of this one, right?

> Implement missing metadata operations
> -
>
> Key: LIVY-575
> URL: https://issues.apache.org/jira/browse/LIVY-575
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Marco Gaido
>Priority: Minor
> Fix For: 0.7.0
>
>
> Many metadata operations (eg. table list retrieval, schema retrieval, ...) 
> are currently not implemented. We should implement them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-07-18 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887692#comment-16887692
 ] 

Yiheng Wang commented on LIVY-575:
--

Hi Marco

I'm interested in contributing to this. There're 11 operations, will submit 
several PRs for it.

Thanks

> Implement missing metadata operations
> -
>
> Key: LIVY-575
> URL: https://issues.apache.org/jira/browse/LIVY-575
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Marco Gaido
>Priority: Minor
>
> Many metadata operations (eg. table list retrieval, schema retrieval, ...) 
> are currently not implemented. We should implement them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-07-18 Thread Yiheng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887786#comment-16887786
 ] 

Yiheng Wang commented on LIVY-575:
--

I can help create sub-tasks

> Implement missing metadata operations
> -
>
> Key: LIVY-575
> URL: https://issues.apache.org/jira/browse/LIVY-575
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Marco Gaido
>Priority: Minor
>
> Many metadata operations (eg. table list retrieval, schema retrieval, ...) 
> are currently not implemented. We should implement them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (LIVY-691) Duplicate jars in assembly jar folder

2019-09-29 Thread Yiheng Wang (Jira)
Yiheng Wang created LIVY-691:


 Summary: Duplicate jars in assembly jar folder
 Key: LIVY-691
 URL: https://issues.apache.org/jira/browse/LIVY-691
 Project: Livy
  Issue Type: Bug
Reporter: Yiheng Wang


We found some jars of different version will be generated both in the assembly 
jar folder. It may cause some problem if there's some API incompatible.

So far we found

-rw-r--r-- 1 root root   966102 Sep 29 14:53 apache-jsp-8.0.33.jar
-rw-r--r-- 1 root root11012 Sep 29 14:53 apache-jsp-9.3.24.v20180605.jar
...
-rw-r--r-- 1 root root38015 Sep 29 14:53 commons-logging-1.0.4.jar
-rw-r--r-- 1 root root61829 Sep 29 14:52 commons-logging-1.2.jar
...
-rw-r--r-- 1 root root   177131 Sep 29 14:52 jetty-util-6.1.26.jar
-rw-r--r-- 1 root root   458642 Sep 29 14:52 jetty-util-9.3.24.v20180605.jar
...
-rw-r--r-- 1 root root50493 Sep 29 14:53 jsp-api-2.0.jar
-rw-r--r-- 1 root root   100636 Sep 29 14:52 jsp-api-2.1.jar
...
-rw-r--r-- 1 root root  1208356 Sep 29 14:53 netty-3.7.0.Final.jar
-rw-r--r-- 1 root root  1330219 Sep 29 14:52 netty-3.9.9.Final.jar



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (LIVY-690) Exclude curator in thrift server pom to avoid conflict jars

2019-09-28 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940216#comment-16940216
 ] 

Yiheng Wang commented on LIVY-690:
--

This will be fixed by this patch:
https://github.com/apache/incubator-livy/pull/239

> Exclude curator in thrift server pom to avoid conflict jars
> ---
>
> Key: LIVY-690
> URL: https://issues.apache.org/jira/browse/LIVY-690
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: Yiheng Wang
>Priority: Major
> Fix For: 0.7.0
>
>
> Currently, thrift server has a dependency of curator-client:2.12.0 through 
> the hive service. After the build, a curator-client-2.12.0.jar file will be 
> generated in the jars folder. It is conflicted with the 
> curator-client-2.7.1.jar file, which is used by livy server.
> We observed that in some JDK, the curator-client-2.12.0.jar is loaded before 
> the curator-client-2.7.1.jar, and will crash the recovery enabled livy server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (LIVY-690) Exclude curator in thrift server pom to avoid conflict jars

2019-09-28 Thread Yiheng Wang (Jira)
Yiheng Wang created LIVY-690:


 Summary: Exclude curator in thrift server pom to avoid conflict 
jars
 Key: LIVY-690
 URL: https://issues.apache.org/jira/browse/LIVY-690
 Project: Livy
  Issue Type: Bug
  Components: Thriftserver
Affects Versions: 0.6.0
Reporter: Yiheng Wang
 Fix For: 0.7.0


Currently, thrift server has a dependency of curator-client:2.12.0 through the 
hive service. After the build, a curator-client-2.12.0.jar file will be 
generated in the jars folder. It is conflicted with the 
curator-client-2.7.1.jar file, which is used by livy server.

We observed that in some JDK, the curator-client-2.12.0.jar is loaded before 
the curator-client-2.7.1.jar, and will crash the recovery enabled livy server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (LIVY-667) Support query a lot of data.

2019-09-23 Thread Yiheng Wang (Jira)


[ 
https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935829#comment-16935829
 ] 

Yiheng Wang commented on LIVY-667:
--

Hi Marco. I think Spark compute on the partition data through an iterator 
interface. The executor may not load the whole partition data into memory.

> Support query a lot of data.
> 
>
> Key: LIVY-667
> URL: https://issues.apache.org/jira/browse/LIVY-667
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator 
> to load one partition at each time instead of the whole rdd to avoid 
> OutOfMemory. However, if the largest partition is too big, the OutOfMemory 
> still occurs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (LIVY-696) Support recovery in livy thrift server

2019-10-10 Thread Yiheng Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/LIVY-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-696:
-
Description: 
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC sessions and operations will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete.

  was:
This JIRA is opened to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC sessions and operations will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete.


> Support recovery in livy thrift server
> --
>
> Key: LIVY-696
> URL: https://issues.apache.org/jira/browse/LIVY-696
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Major
>
> This JIRA is to discuss support recovery in Livy Thrift server.
> Livy server support recovery. If user set *livy.server.recovery.mode* to 
> *recovery* and set the *state-store*, livy server will persist its states. 
> Once livy server crashes, the sessions will be restored after the service 
> restart.
> The recovery is not yet supported in thrift server. After the service 
> restarts, all JDBC sessions and operations will be lost.
> Should we support recovery in the thrift server? There's one concern. In JDBC 
> binary mode,  the connection will be lost if the server crashes, and looks 
> like hive-jdbc cannot reuse session-id when reconnecting to the server. But 
> in JDBC http mode, we will benefit from the recovery. As http mode use short 
> http connections instead of long connection.
> There're also two levels of recovery need to be considered.
> *Session recovery* let user can still use their JDBC connection after service 
> restart. It should be straight forward to implement. We just need to persist 
> the JDBC session/livy session mapping to the state-store and recover the 
> mapping when restarting.
> *Operation recovery* let user can still use the JDBC statement(or fetch 
> result) after service restart. This needs to persist 
> ExecuteStatementOperation state. The concern is some states are not suitable 
> to persist to state-store, e.g. operationMessages, rowOffset, state, 
> operationException, backgroundHandle, lastAccessTime, operationComplete.



--
This message was sent by 

[jira] [Updated] (LIVY-696) Support recovery in livy thrift server

2019-10-10 Thread Yiheng Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/LIVY-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-696:
-
Description: 
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC connections and statements will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete.

  was:
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC sessions and operations will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete.


> Support recovery in livy thrift server
> --
>
> Key: LIVY-696
> URL: https://issues.apache.org/jira/browse/LIVY-696
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Major
>
> This JIRA is to discuss support recovery in Livy Thrift server.
> Livy server support recovery. If user set *livy.server.recovery.mode* to 
> *recovery* and set the *state-store*, livy server will persist its states. 
> Once livy server crashes, the sessions will be restored after the service 
> restart.
> The recovery is not yet supported in thrift server. After the service 
> restarts, all JDBC connections and statements will be lost.
> Should we support recovery in the thrift server? There's one concern. In JDBC 
> binary mode,  the connection will be lost if the server crashes, and looks 
> like hive-jdbc cannot reuse session-id when reconnecting to the server. But 
> in JDBC http mode, we will benefit from the recovery. As http mode use short 
> http connections instead of long connection.
> There're also two levels of recovery need to be considered.
> *Session recovery* let user can still use their JDBC connection after service 
> restart. It should be straight forward to implement. We just need to persist 
> the JDBC session/livy session mapping to the state-store and recover the 
> mapping when restarting.
> *Operation recovery* let user can still use the JDBC statement(or fetch 
> result) after service restart. This needs to persist 
> ExecuteStatementOperation state. The concern is some states are not suitable 
> to persist to state-store, e.g. operationMessages, rowOffset, state, 
> operationException, backgroundHandle, lastAccessTime, operationComplete.



--
This message was sent by 

[jira] [Updated] (LIVY-696) Support recovery in livy thrift server

2019-10-10 Thread Yiheng Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/LIVY-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-696:
-
Description: 
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC connections and statements will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete. They're complex type or change frequently.

  was:
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC connections and statements will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete.


> Support recovery in livy thrift server
> --
>
> Key: LIVY-696
> URL: https://issues.apache.org/jira/browse/LIVY-696
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Major
>
> This JIRA is to discuss support recovery in Livy Thrift server.
> Livy server support recovery. If user set *livy.server.recovery.mode* to 
> *recovery* and set the *state-store*, livy server will persist its states. 
> Once livy server crashes, the sessions will be restored after the service 
> restart.
> The recovery is not yet supported in thrift server. After the service 
> restarts, all JDBC connections and statements will be lost.
> Should we support recovery in the thrift server? There's one concern. In JDBC 
> binary mode,  the connection will be lost if the server crashes, and looks 
> like hive-jdbc cannot reuse session-id when reconnecting to the server. But 
> in JDBC http mode, we will benefit from the recovery. As http mode use short 
> http connections instead of long connection.
> There're also two levels of recovery need to be considered.
> *Session recovery* let user can still use their JDBC connection after service 
> restart. It should be straight forward to implement. We just need to persist 
> the JDBC session/livy session mapping to the state-store and recover the 
> mapping when restarting.
> *Operation recovery* let user can still use the JDBC statement(or fetch 
> result) after service restart. This needs to persist 
> ExecuteStatementOperation state. The concern is some states are not suitable 
> to persist to state-store, e.g. operationMessages, rowOffset, state, 
> operationException, backgroundHandle, lastAccessTime, 

[jira] [Created] (LIVY-696) Support recovery in livy thrift server

2019-10-10 Thread Yiheng Wang (Jira)
Yiheng Wang created LIVY-696:


 Summary: Support recovery in livy thrift server
 Key: LIVY-696
 URL: https://issues.apache.org/jira/browse/LIVY-696
 Project: Livy
  Issue Type: Improvement
  Components: Thriftserver
Reporter: Yiheng Wang


This JIRA is opened to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC sessions and operations will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (LIVY-696) Support recovery in livy thrift server

2019-10-10 Thread Yiheng Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/LIVY-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiheng Wang updated LIVY-696:
-
Description: 
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC connections and statements will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Statement recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete. They're complex type or change frequently.

  was:
This JIRA is to discuss support recovery in Livy Thrift server.

Livy server support recovery. If user set *livy.server.recovery.mode* to 
*recovery* and set the *state-store*, livy server will persist its states. Once 
livy server crashes, the sessions will be restored after the service restart.

The recovery is not yet supported in thrift server. After the service restarts, 
all JDBC connections and statements will be lost.

Should we support recovery in the thrift server? There's one concern. In JDBC 
binary mode,  the connection will be lost if the server crashes, and looks like 
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC 
http mode, we will benefit from the recovery. As http mode use short http 
connections instead of long connection.

There're also two levels of recovery need to be considered.

*Session recovery* let user can still use their JDBC connection after service 
restart. It should be straight forward to implement. We just need to persist 
the JDBC session/livy session mapping to the state-store and recover the 
mapping when restarting.

*Operation recovery* let user can still use the JDBC statement(or fetch result) 
after service restart. This needs to persist ExecuteStatementOperation state. 
The concern is some states are not suitable to persist to state-store, e.g. 
operationMessages, rowOffset, state, operationException, backgroundHandle, 
lastAccessTime, operationComplete. They're complex type or change frequently.


> Support recovery in livy thrift server
> --
>
> Key: LIVY-696
> URL: https://issues.apache.org/jira/browse/LIVY-696
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Yiheng Wang
>Priority: Major
>
> This JIRA is to discuss support recovery in Livy Thrift server.
> Livy server support recovery. If user set *livy.server.recovery.mode* to 
> *recovery* and set the *state-store*, livy server will persist its states. 
> Once livy server crashes, the sessions will be restored after the service 
> restart.
> The recovery is not yet supported in thrift server. After the service 
> restarts, all JDBC connections and statements will be lost.
> Should we support recovery in the thrift server? There's one concern. In JDBC 
> binary mode,  the connection will be lost if the server crashes, and looks 
> like hive-jdbc cannot reuse session-id when reconnecting to the server. But 
> in JDBC http mode, we will benefit from the recovery. As http mode use short 
> http connections instead of long connection.
> There're also two levels of recovery need to be considered.
> *Session recovery* let user can still use their JDBC connection after service 
> restart. It should be straight forward to implement. We just need to persist 
> the JDBC session/livy session mapping to the state-store and recover the 
> mapping when restarting.
> *Statement recovery* let user can still use the JDBC statement(or fetch 
> result) after service restart. This needs to persist 
> ExecuteStatementOperation state. The concern is some states are not suitable 
> to persist to state-store, e.g. operationMessages, rowOffset, state, 
> operationException,