[jira] [Commented] (LENS-1326) session/sessions API returning no data on GUI/API though there are active sessions

2016-09-30 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/LENS-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535998#comment-15535998
 ] 

Hadoop QA commented on LENS-1326:
-

Applied patch: 
[getsession-error.patch|https://issues.apache.org/jira/secure/attachment/12831101/getsession-error.patch]
 and ran command: mvn clean install -fae. Result: Success. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/966/

> session/sessions API returning no data on GUI/API though there are active 
> sessions
> --
>
> Key: LENS-1326
> URL: https://issues.apache.org/jira/browse/LENS-1326
> Project: Apache Lens
>  Issue Type: Bug
>Reporter: Puneet Gupta
>Assignee: Lavkesh Lahngir
> Attachments: getsession-error.patch
>
>
> The lens GUI and API is showing no sessions via sessions REST end point.
> Server metrics are showing active sessions  though. 
> {noformat}
> "org.apache.lens.server.api.session.SessionService.active-sessions" : {
>  "value" : 5324
>},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52040: Handling data delay

2016-09-30 Thread Lavkesh Lahngir 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52040/
---

(Updated Sept. 30, 2016, 1 p.m.)


Review request for lens.


Bugs: LENS-1327
https://issues.apache.org/jira/browse/LENS-1327


Repository: lens


Description
---

A instance goes to waiting state when there is some data missing. They are 
retried after some time.


Diffs (updated)
-

  
lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceEvent.java
 4f3409c21748b2ae026bdeba881dcc5a988a2d3c 
  
lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceState.java
 3d2605ea67c99bdba9e56140e0e13c9685a19c7a 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
8f1983ee10e9e4a429dc1c8ceb5d4a65bab975c1 
  lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerDAO.java 
b924167fb819b3baae6be14efbb085fe72f99f54 
  
lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerEventListener.java
 1db8136c4e352d7dc03d7614511e89c6209ae151 
  
lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerServiceImpl.java
 969d7403ddc0cb2a8bfa59ffde64ea249714a08e 
  lens-server/src/main/resources/lensserver-default.xml 
84ff9f9e42015da570eae94bf9748aa090417bae 
  
lens-server/src/test/java/org/apache/lens/server/scheduler/SchedulerDAOTest.java
 16a6c78ba5b36e3a164646fec6c2aca9f032d03e 
  
lens-server/src/test/java/org/apache/lens/server/scheduler/SchedulerRestartTest.java
 de364994ebf70ba92692f1745bf254b1c0046d79 
  src/site/apt/admin/config.apt 24d5fd6e9bcfc38b6543ffc8ac16a32a025d70c5 

Diff: https://reviews.apache.org/r/52040/diff/


Testing
---


Thanks,

Lavkesh Lahngir

Re: Review Request 52088: LENS-743: Query failure retries for transient errors

2016-09-30 Thread Rajat Khandelwal 


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > checkstyle/src/main/resources/checkstyle.xml, line 65
> > 
> >
> > hmm, what file is extending this? Need to fix by doing some refactoring 
> > if required.

Query execution service impl is now 3558 lines. Will see if any refactoring can 
be done.


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java, lines 
> > 303-304
> > 
> >
> > Adding resources specific to driver is removed?

This is being done in super class's method.


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, 
> > line 365
> > 
> >
> > hiveConf would be required here.

Adding overlay in the next line. `super.configure` is supposed to add driver 
specific configurations and this can be asked by `getConf()`


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, 
> > line 368
> > 
> >
> > hivedriver-site.xml is not added in old code.

Refer to the above comment, super class making sure `getConf()` returns driver 
specific configuration. And this line is to make sure the overlay takes effect.


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java,
> >  lines 76-77
> > 
> >
> > This is enforcing drivertype in the file names. This will be a problem 
> > with -default.xml files, if people create driver type to be different than 
> > whats expected to be.
> > 
> > Instead you can try using driver.getClass().getName().toLowercase() ?

I'm fine with either, but unable to see the merit in using one above the other.


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java,
> >  line 82
> > 
> >
> > removing from context map? We might lose some details like plan and 
> > etc. May be should be moved another data structure

Those things will anyway get lost if the query succeeds on another driver. If 
there are no other drivers, this is restored.


> On Sept. 30, 2016, 12:16 p.m., Amareshwari Sriramadasu wrote:
> > lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java, line 113
> > 
> >
> > can you update state machine on user doc with the change?

Would need to generate the png file again.


- Rajat


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52088/#review150991
---


On Sept. 29, 2016, 2:39 p.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52088/
> ---
> 
> (Updated Sept. 29, 2016, 2:39 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-743
> https://issues.apache.org/jira/browse/LENS-743
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> There have to be retries for query failures for transient errors like network 
> errors (Hive server not reachable/ Metastore not reachable/ DB not 
> reachable). Retries should be available for each phase - submission, 
> execution, updating status, fetching results and formatting.
> 
> Right now, any such failure results in marking query as failed.
> 
> 
> Diffs
> -
> 
>   checkstyle/src/main/resources/checkstyle.xml 
> 17e55ea1071c06c73e8bf387cd69acca83bc 
>   lens-api/src/main/java/org/apache/lens/api/query/FailedAttempt.java 
> PRE-CREATION 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 
> 7d079a9ed99e3d4940d5f26370547683a6bba465 
>   lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java 
> daedbf9ea387cbde3ef39363c2beef3f96725223 
>   
> lens-api/src/test/java/org/apache/lens/api/jaxb/YAMLToStringStrategyTest.java 
> 04d9ad6dee942c705b0cdf071f49f1305696da65 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.LensQuery.xml 
> a681bca7fd6e46fe992901e04e025a8a3106 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.QueryStatus.xml
>  70f9c932efc090095273566e

Re: Review Request 52088: LENS-743: Query failure retries for transient errors

2016-09-30 Thread Rajat Khandelwal 


> On Sept. 30, 2016, 12:34 p.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/query/StatusUpdateFailureContext.java,
> >  line 41
> > 
> >
> > why do we need clear() functionality ?

The file FailureContext is renamed to `StatusUpdateFailureContext`. It already 
had `clear` method. The method isn't propagated to the interface, but is 
retained in the implementation. The other implementation of the interface 
doesn't have a `clear` method. 

And to answer your question, it's used in 
`QueryContext#updateDriverStatus(BackOffRetryHandler)`.


> On Sept. 30, 2016, 12:34 p.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/query/events/PriorityChange.java,
> >  line 27
> > 
> >
> > when can priority of the query can change? and why?

Again, pre-existing code, only moved to another package. AFAIK, it's not being 
used anywhere.


> On Sept. 30, 2016, 12:34 p.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java,
> >  line 385
> > 
> >
> > loadDrivers() called separately?

Yes. The method `loadDriversAndSelectors` was basically a bunch of code and a 
call to `loadSelectors`. So extracted that bunch of code into a method 
`loadDrivers`. Both are called sequencially in `init`


> On Sept. 30, 2016, 12:34 p.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java,
> >  line 905
> > 
> >
> > Would need re-run for jdbc queries, as connection would have closed.

I don't think any additional code is required for that. query status will be 
updated to FAILING, and then retry policy will be consulted. The retry policy 
should be taking care of forcing a retry in case a failure is due to connection 
close.


- Rajat


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52088/#review150993
---


On Sept. 29, 2016, 2:39 p.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52088/
> ---
> 
> (Updated Sept. 29, 2016, 2:39 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-743
> https://issues.apache.org/jira/browse/LENS-743
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> There have to be retries for query failures for transient errors like network 
> errors (Hive server not reachable/ Metastore not reachable/ DB not 
> reachable). Retries should be available for each phase - submission, 
> execution, updating status, fetching results and formatting.
> 
> Right now, any such failure results in marking query as failed.
> 
> 
> Diffs
> -
> 
>   checkstyle/src/main/resources/checkstyle.xml 
> 17e55ea1071c06c73e8bf387cd69acca83bc 
>   lens-api/src/main/java/org/apache/lens/api/query/FailedAttempt.java 
> PRE-CREATION 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 
> 7d079a9ed99e3d4940d5f26370547683a6bba465 
>   lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java 
> daedbf9ea387cbde3ef39363c2beef3f96725223 
>   
> lens-api/src/test/java/org/apache/lens/api/jaxb/YAMLToStringStrategyTest.java 
> 04d9ad6dee942c705b0cdf071f49f1305696da65 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.LensQuery.xml 
> a681bca7fd6e46fe992901e04e025a8a3106 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.QueryStatus.xml
>  70f9c932efc090095273566eaa52d380a2f23842 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> fceabeae9217676774a6978212cbc9ee9ca120f7 
>   lens-driver-es/src/test/resources/hive-site.xml 
> e49c876289d5352203d2523ddb9134d745c197e5 
>   lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
> 84d9933da506e3f1bff2d3e0ea403ed1edf0fe35 
>   lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
> 3bf5e8f7dc9d032e46d06c3245048096acb4572e 
>   
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriverConfConstants.java
>  3c34eb191966dff3ac3032eb79282336b5c6789b 
>   
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java
>  82b56470e2de09f7c41d1226ea59cffdd8d26a26 
>   
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java
>  6e9086f9e8dc302ad2171d7f5636a24cb79dcb50 
>   
> lens-server-api/src/main/jav

Re: Review Request 52088: LENS-743: Query failure retries for transient errors

2016-09-30 Thread Rajat Khandelwal 


> On Sept. 22, 2016, 11:53 a.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/common/SingleRetryHandler.java,
> >  line 22
> > 
> >
> > Will SingleRetryHandler have delay for the next try?
> 
> Rajat Khandelwal wrote:
> Will add more policies. This is one of the simplest of policies which has 
> only one retry, without any delay.

Have generalized the policy to call it ImmediateRetryPolicy. The constructor 
takes an integer argument which is the number of retries.


> On Sept. 22, 2016, 11:53 a.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java,
> >  lines 210-216
> > 
> >
> > All these should be defined as transient? Or there is recovery upon 
> > restarts?
> 
> Rajat Khandelwal wrote:
> The failed attempts will have recovery upon restart
> 
> For retry policies, they need not be stored, but it might result in a 
> query being retried more than the normal amount, in case the query failed and 
> is going through a retry and fails again on restart. Then the retry cycle 
> will restart, causing the retries happened before restart to become 
> redundant. Which should be fine assuming retries will not be that frequent.

Upon second thuoght, I've kept all these non-transient. These will be persisted 
and recovered upon restart, which will maintain consistencey of retries across 
restart. Have added a test case for the same.


- Rajat


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52088/#review149955
---


On Sept. 29, 2016, 2:39 p.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52088/
> ---
> 
> (Updated Sept. 29, 2016, 2:39 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-743
> https://issues.apache.org/jira/browse/LENS-743
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> There have to be retries for query failures for transient errors like network 
> errors (Hive server not reachable/ Metastore not reachable/ DB not 
> reachable). Retries should be available for each phase - submission, 
> execution, updating status, fetching results and formatting.
> 
> Right now, any such failure results in marking query as failed.
> 
> 
> Diffs
> -
> 
>   checkstyle/src/main/resources/checkstyle.xml 
> 17e55ea1071c06c73e8bf387cd69acca83bc 
>   lens-api/src/main/java/org/apache/lens/api/query/FailedAttempt.java 
> PRE-CREATION 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 
> 7d079a9ed99e3d4940d5f26370547683a6bba465 
>   lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java 
> daedbf9ea387cbde3ef39363c2beef3f96725223 
>   
> lens-api/src/test/java/org/apache/lens/api/jaxb/YAMLToStringStrategyTest.java 
> 04d9ad6dee942c705b0cdf071f49f1305696da65 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.LensQuery.xml 
> a681bca7fd6e46fe992901e04e025a8a3106 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.QueryStatus.xml
>  70f9c932efc090095273566eaa52d380a2f23842 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> fceabeae9217676774a6978212cbc9ee9ca120f7 
>   lens-driver-es/src/test/resources/hive-site.xml 
> e49c876289d5352203d2523ddb9134d745c197e5 
>   lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
> 84d9933da506e3f1bff2d3e0ea403ed1edf0fe35 
>   lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
> 3bf5e8f7dc9d032e46d06c3245048096acb4572e 
>   
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriverConfConstants.java
>  3c34eb191966dff3ac3032eb79282336b5c6789b 
>   
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java
>  82b56470e2de09f7c41d1226ea59cffdd8d26a26 
>   
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java
>  6e9086f9e8dc302ad2171d7f5636a24cb79dcb50 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  8cf617b7181b9a305ea2c68f68c610f823ee75d7 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/BackOffRetryHandler.java
>  17bfba02244bca7141268ed12e4a41400e2bec82 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/FailureContext.java
>  70a34b0c0517ed3b7af46120906fb77ae57870ac 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/FibonacciExponentialBackOffRetryHandler.java
>  e7fb8ce0194803317aaa0e1c5f4e3e0fb6bbca8c

Re: Review Request 51363: LENS-1287: Create command in cli to create schema taking parent directory where schema is stored

2016-09-30 Thread Rajat Khandelwal 


> On Sept. 22, 2016, 11:20 a.m., Amareshwari Sriramadasu wrote:
> > lens-cli/src/main/java/org/apache/lens/cli/commands/LensSchemaCommands.java,
> >  line 94
> > 
> >
> > Can we this message come on cli console as well ?

This log will come on CLI. This isn't slf4j logger, this is shell's logger.


- Rajat


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51363/#review149949
---


On Sept. 30, 2016, 5:50 p.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51363/
> ---
> 
> (Updated Sept. 30, 2016, 5:50 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1287
> https://issues.apache.org/jira/browse/LENS-1287
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java 
> 05108f0158b655653f0665da80a2d10b24f33233 
>   
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java
>  7df5c7ca1c7f557409ed36a4be7f4b050260f159 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensFactCommands.java 
> 6457bceef143ce08e6a57726cb1e54dcf3288b0c 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensSchemaCommands.java 
> PRE-CREATION 
>   lens-cli/src/test/java/org/apache/lens/cli/LensCLITest.java PRE-CREATION 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensCubeCommands.java 
> c20766c26c42918e7ab33c32c0b4218ddfa5ee81 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 
> 9687c6089125b4eb9dc2a96fd7d4a140c81eb66c 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensDimensionCommands.java 
> a3f01c38d1c94857f0cdfdd7a457b402754c30a6 
>   
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDimensionTableCommands.java
>  5d6d7682d75793897cc6d2fee0bb23ce7f467ab9 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensFactCommands.java 
> b4cbf344c46faa40074bc79fa0227312e027dfb6 
>   
> lens-cli/src/test/java/org/apache/lens/cli/TestLensFactCommandsWithMissingWeight.java
>  24f9279b24de98ed1e7304ec2e2010c17df9f8d8 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java 
> e75fc0ef19d9d9b288bdb677bfdf499aa088e6ee 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensSchemaCommands.java 
> PRE-CREATION 
>   
> lens-cli/src/test/java/org/apache/lens/cli/TestLensSegmentationCommands.java 
> e6d4bc5354bbcae9ea6d1f06ec2be6e2ff885cf6 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensStorageCommands.java 
> c4ab6140c573d8bb4d2048969be14786d2371d6d 
>   lens-cli/src/test/java/org/apache/lens/cli/doc/TestGenerateCLIUserDoc.java 
> 95e6b4e205128a19d1099f0a14638a8eb2d19c7c 
>   lens-cli/src/test/resources/cube_with_no_weight_facts.xml 
> 4673ca416758c35bf50616896f084360a1b5bb89 
>   lens-cli/src/test/resources/dim_table.xml 
> eb4ddfcc49c9cfdc152df35fe863176986016d95 
>   lens-cli/src/test/resources/dim_table2.xml 
> 8440e28b59cdcc567bac00e4c1e2367ef2ebda0c 
>   lens-cli/src/test/resources/fact1.xml 
> b18a39313bff9850c720cb3f0492bc6b7a650f78 
>   lens-cli/src/test/resources/fact_without_weight.xml 
> 0c124b2aba88a9efcc164b3b4de96479c164cd6a 
>   lens-cli/src/test/resources/local-storage.xml 
> 65513757f5dc2a19fea88cde4f4fa8736f814091 
>   lens-cli/src/test/resources/sample-cube.xml 
> 2d6bc3469466cee6245f8f503af956dcc92cffb5 
>   lens-cli/src/test/resources/schema/cubes/base/cube_with_no_weight_facts.xml 
> PRE-CREATION 
>   lens-cli/src/test/resources/schema/cubes/base/sample-cube.xml PRE-CREATION 
>   lens-cli/src/test/resources/schema/dimensions/test-detail.xml PRE-CREATION 
>   lens-cli/src/test/resources/schema/dimensions/test-dimension.xml 
> PRE-CREATION 
>   lens-cli/src/test/resources/schema/dimtables/dim_table.xml PRE-CREATION 
>   lens-cli/src/test/resources/schema/dimtables/dim_table2.xml PRE-CREATION 
>   lens-cli/src/test/resources/schema/facts/fact1.xml PRE-CREATION 
>   lens-cli/src/test/resources/schema/facts/fact_without_weight.xml 
> PRE-CREATION 
>   lens-cli/src/test/resources/schema/segmentations/seg1.xml PRE-CREATION 
>   lens-cli/src/test/resources/schema/storages/local-storage.xml PRE-CREATION 
>   lens-cli/src/test/resources/seg1.xml 
> 8a3991514633bcb25ded0e305bdd50192149773c 
>   lens-cli/src/test/resources/test-detail.xml 
> b51c1888d88436302a7e43ef12d3643fe99b0267 
>   lens-cli/src/test/resources/test-dimension.xml 
> 01de8e69bce7536768bd99d765af3d07f70914a4 
>   src/site/apt/user/cli.apt 2522e03fc80b12bcd64f6df155cd862e4e53e2c2 
> 
> Diff: https://reviews.apache.org/r/51363/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

Re: Review Request 51363: LENS-1287: Create command in cli to create schema taking parent directory where schema is stored

2016-09-30 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51363/
---

(Updated Sept. 30, 2016, 5:50 p.m.)


Review request for lens.


Bugs: LENS-1287
https://issues.apache.org/jira/browse/LENS-1287


Repository: lens


Description
---


Diffs (updated)
-

  lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java 
05108f0158b655653f0665da80a2d10b24f33233 
  
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java
 7df5c7ca1c7f557409ed36a4be7f4b050260f159 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensFactCommands.java 
6457bceef143ce08e6a57726cb1e54dcf3288b0c 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensSchemaCommands.java 
PRE-CREATION 
  lens-cli/src/test/java/org/apache/lens/cli/LensCLITest.java PRE-CREATION 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensCubeCommands.java 
c20766c26c42918e7ab33c32c0b4218ddfa5ee81 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 
9687c6089125b4eb9dc2a96fd7d4a140c81eb66c 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensDimensionCommands.java 
a3f01c38d1c94857f0cdfdd7a457b402754c30a6 
  
lens-cli/src/test/java/org/apache/lens/cli/TestLensDimensionTableCommands.java 
5d6d7682d75793897cc6d2fee0bb23ce7f467ab9 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensFactCommands.java 
b4cbf344c46faa40074bc79fa0227312e027dfb6 
  
lens-cli/src/test/java/org/apache/lens/cli/TestLensFactCommandsWithMissingWeight.java
 24f9279b24de98ed1e7304ec2e2010c17df9f8d8 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java 
e75fc0ef19d9d9b288bdb677bfdf499aa088e6ee 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensSchemaCommands.java 
PRE-CREATION 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensSegmentationCommands.java 
e6d4bc5354bbcae9ea6d1f06ec2be6e2ff885cf6 
  lens-cli/src/test/java/org/apache/lens/cli/TestLensStorageCommands.java 
c4ab6140c573d8bb4d2048969be14786d2371d6d 
  lens-cli/src/test/java/org/apache/lens/cli/doc/TestGenerateCLIUserDoc.java 
95e6b4e205128a19d1099f0a14638a8eb2d19c7c 
  lens-cli/src/test/resources/cube_with_no_weight_facts.xml 
4673ca416758c35bf50616896f084360a1b5bb89 
  lens-cli/src/test/resources/dim_table.xml 
eb4ddfcc49c9cfdc152df35fe863176986016d95 
  lens-cli/src/test/resources/dim_table2.xml 
8440e28b59cdcc567bac00e4c1e2367ef2ebda0c 
  lens-cli/src/test/resources/fact1.xml 
b18a39313bff9850c720cb3f0492bc6b7a650f78 
  lens-cli/src/test/resources/fact_without_weight.xml 
0c124b2aba88a9efcc164b3b4de96479c164cd6a 
  lens-cli/src/test/resources/local-storage.xml 
65513757f5dc2a19fea88cde4f4fa8736f814091 
  lens-cli/src/test/resources/sample-cube.xml 
2d6bc3469466cee6245f8f503af956dcc92cffb5 
  lens-cli/src/test/resources/schema/cubes/base/cube_with_no_weight_facts.xml 
PRE-CREATION 
  lens-cli/src/test/resources/schema/cubes/base/sample-cube.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/dimensions/test-detail.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/dimensions/test-dimension.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/dimtables/dim_table.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/dimtables/dim_table2.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/facts/fact1.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/facts/fact_without_weight.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/segmentations/seg1.xml PRE-CREATION 
  lens-cli/src/test/resources/schema/storages/local-storage.xml PRE-CREATION 
  lens-cli/src/test/resources/seg1.xml 8a3991514633bcb25ded0e305bdd50192149773c 
  lens-cli/src/test/resources/test-detail.xml 
b51c1888d88436302a7e43ef12d3643fe99b0267 
  lens-cli/src/test/resources/test-dimension.xml 
01de8e69bce7536768bd99d765af3d07f70914a4 
  src/site/apt/user/cli.apt 2522e03fc80b12bcd64f6df155cd862e4e53e2c2 

Diff: https://reviews.apache.org/r/51363/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Updated] (LENS-1326) session/sessions API returning no data on GUI/API though there are active sessions

2016-09-30 Thread Lavkesh Lahngir (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LENS-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lavkesh Lahngir updated LENS-1326:
--
Status: Patch Available  (was: Open)

> session/sessions API returning no data on GUI/API though there are active 
> sessions
> --
>
> Key: LENS-1326
> URL: https://issues.apache.org/jira/browse/LENS-1326
> Project: Apache Lens
>  Issue Type: Bug
>Reporter: Puneet Gupta
>Assignee: Lavkesh Lahngir
> Attachments: getsession-error.patch
>
>
> The lens GUI and API is showing no sessions via sessions REST end point.
> Server metrics are showing active sessions  though. 
> {noformat}
> "org.apache.lens.server.api.session.SessionService.active-sessions" : {
>  "value" : 5324
>},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-743) Query failure retries for transient errors

2016-09-30 Thread Rajat Khandelwal (JIRA) 

[ 
https://issues.apache.org/jira/browse/LENS-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535812#comment-15535812
 ] 

Rajat Khandelwal commented on LENS-743:
---

The policies and retries are persisted and recovered upon restart. So it 
resumes from where it left off. So if retries are not exhausted and query is in 
queue, it'll be restored in the queue, from where QuerySubmitter takes care of 
taking retry policy into consideration. 

> Query failure retries for transient errors
> --
>
> Key: LENS-743
> URL: https://issues.apache.org/jira/browse/LENS-743
> Project: Apache Lens
>  Issue Type: Improvement
>  Components: server
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
>  Labels: gsoc2016, java
>
> There have to be retries for query failures for transient errors like network 
> errors (Hive server not reachable/ Metastore not reachable/ DB not 
> reachable). Retries should be available for each phase - submission, 
> execution, updating status, fetching results and formatting.
> Right now, any such failure results in marking query as failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1326) session/sessions API returning no data on GUI/API though there are active sessions

2016-09-30 Thread Lavkesh Lahngir (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LENS-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lavkesh Lahngir updated LENS-1326:
--
Attachment: getsession-error.patch

> session/sessions API returning no data on GUI/API though there are active 
> sessions
> --
>
> Key: LENS-1326
> URL: https://issues.apache.org/jira/browse/LENS-1326
> Project: Apache Lens
>  Issue Type: Bug
>Reporter: Puneet Gupta
>Assignee: Lavkesh Lahngir
> Attachments: getsession-error.patch
>
>
> The lens GUI and API is showing no sessions via sessions REST end point.
> Server metrics are showing active sessions  though. 
> {noformat}
> "org.apache.lens.server.api.session.SessionService.active-sessions" : {
>  "value" : 5324
>},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52040: Handling data delay

2016-09-30 Thread Amareshwari Sriramadasu 


> On Sept. 29, 2016, 4:58 a.m., Amareshwari Sriramadasu wrote:
> > lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceState.java,
> >  line 60
> > 
> >
> > I'm still wondering if there should be a LAUNCHED state for 
> > ON_CONDITIONS_MET.
> 
> Lavkesh Lahngir wrote:
> That's why I changed it to LAUNCHING state from LAUNCHED.

LAUNCHING->WAITING->LAUNCHING->WAITING->LAUNCHING->WAITING->LAUNCHING and 
finally stopping at LAUNCHING only.

should we make  
LAUNCHING->WAITING->LAUNCHING->WAITING->LAUNCHING->WAITING->LAUNCHING-LAUNCHED 
? it would be clean state machine. Otherwise it seems confusing.

LAUNCHED is when all conditions are met. Then it is given to query service - it 
will be in LAUNCHED state as long as it is queued in query service. becomes 
running when it is running


- Amareshwari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52040/#review150810
---


On Sept. 26, 2016, 9:53 a.m., Lavkesh Lahngir wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52040/
> ---
> 
> (Updated Sept. 26, 2016, 9:53 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1327
> https://issues.apache.org/jira/browse/LENS-1327
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> A instance goes to waiting state when there is some data missing. They are 
> retried after some time.
> 
> 
> Diffs
> -
> 
>   
> lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceEvent.java
>  4f3409c21748b2ae026bdeba881dcc5a988a2d3c 
>   
> lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceState.java
>  3d2605ea67c99bdba9e56140e0e13c9685a19c7a 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  8f1983ee10e9e4a429dc1c8ceb5d4a65bab975c1 
>   
> lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerDAO.java 
> b924167fb819b3baae6be14efbb085fe72f99f54 
>   
> lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerEventListener.java
>  1db8136c4e352d7dc03d7614511e89c6209ae151 
>   
> lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerServiceImpl.java
>  969d7403ddc0cb2a8bfa59ffde64ea249714a08e 
>   lens-server/src/main/resources/lensserver-default.xml 
> 84ff9f9e42015da570eae94bf9748aa090417bae 
>   
> lens-server/src/test/java/org/apache/lens/server/scheduler/SchedulerDAOTest.java
>  16a6c78ba5b36e3a164646fec6c2aca9f032d03e 
>   src/site/apt/admin/config.apt 24d5fd6e9bcfc38b6543ffc8ac16a32a025d70c5 
> 
> Diff: https://reviews.apache.org/r/52040/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Lavkesh Lahngir
> 
>

Re: Review Request 52040: Handling data delay

2016-09-30 Thread Lavkesh Lahngir 


> On Sept. 29, 2016, 4:58 a.m., Amareshwari Sriramadasu wrote:
> > lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceState.java,
> >  line 60
> > 
> >
> > I'm still wondering if there should be a LAUNCHED state for 
> > ON_CONDITIONS_MET.

That's why I changed it to LAUNCHING state from LAUNCHED.


- Lavkesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52040/#review150810
---


On Sept. 26, 2016, 9:53 a.m., Lavkesh Lahngir wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52040/
> ---
> 
> (Updated Sept. 26, 2016, 9:53 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1327
> https://issues.apache.org/jira/browse/LENS-1327
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> A instance goes to waiting state when there is some data missing. They are 
> retried after some time.
> 
> 
> Diffs
> -
> 
>   
> lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceEvent.java
>  4f3409c21748b2ae026bdeba881dcc5a988a2d3c 
>   
> lens-api/src/main/java/org/apache/lens/api/scheduler/SchedulerJobInstanceState.java
>  3d2605ea67c99bdba9e56140e0e13c9685a19c7a 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  8f1983ee10e9e4a429dc1c8ceb5d4a65bab975c1 
>   
> lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerDAO.java 
> b924167fb819b3baae6be14efbb085fe72f99f54 
>   
> lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerEventListener.java
>  1db8136c4e352d7dc03d7614511e89c6209ae151 
>   
> lens-server/src/main/java/org/apache/lens/server/scheduler/SchedulerServiceImpl.java
>  969d7403ddc0cb2a8bfa59ffde64ea249714a08e 
>   lens-server/src/main/resources/lensserver-default.xml 
> 84ff9f9e42015da570eae94bf9748aa090417bae 
>   
> lens-server/src/test/java/org/apache/lens/server/scheduler/SchedulerDAOTest.java
>  16a6c78ba5b36e3a164646fec6c2aca9f032d03e 
>   src/site/apt/admin/config.apt 24d5fd6e9bcfc38b6543ffc8ac16a32a025d70c5 
> 
> Diff: https://reviews.apache.org/r/52040/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Lavkesh Lahngir
> 
>

[jira] [Commented] (LENS-743) Query failure retries for transient errors

2016-09-30 Thread Amareshwari Sriramadasu (JIRA) 

[ 
https://issues.apache.org/jira/browse/LENS-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535281#comment-15535281
 ] 

Amareshwari Sriramadasu commented on LENS-743:
--

[~prongs] Can you also update (how?) the flow changes if server restarts in 
between a query just failed without exhausting retries and  more cases ?

> Query failure retries for transient errors
> --
>
> Key: LENS-743
> URL: https://issues.apache.org/jira/browse/LENS-743
> Project: Apache Lens
>  Issue Type: Improvement
>  Components: server
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
>  Labels: gsoc2016, java
>
> There have to be retries for query failures for transient errors like network 
> errors (Hive server not reachable/ Metastore not reachable/ DB not 
> reachable). Retries should be available for each phase - submission, 
> execution, updating status, fetching results and formatting.
> Right now, any such failure results in marking query as failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52088: LENS-743: Query failure retries for transient errors

2016-09-30 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52088/#review150993
---




lens-server-api/src/main/java/org/apache/lens/server/api/query/StatusUpdateFailureContext.java
 (line 41)


why do we need clear() functionality ?



lens-server-api/src/main/java/org/apache/lens/server/api/query/events/PriorityChange.java
 (line 27)


when can priority of the query can change? and why?



lens-server-api/src/main/java/org/apache/lens/server/api/retry/FailureContext.java
 (line 25)


Add javadoc for all the methods.



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 


loadDrivers() called separately?



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 900)


Would need re-run for jdbc queries, as connection would have closed.


- Amareshwari Sriramadasu


On Sept. 29, 2016, 9:09 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52088/
> ---
> 
> (Updated Sept. 29, 2016, 9:09 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-743
> https://issues.apache.org/jira/browse/LENS-743
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> There have to be retries for query failures for transient errors like network 
> errors (Hive server not reachable/ Metastore not reachable/ DB not 
> reachable). Retries should be available for each phase - submission, 
> execution, updating status, fetching results and formatting.
> 
> Right now, any such failure results in marking query as failed.
> 
> 
> Diffs
> -
> 
>   checkstyle/src/main/resources/checkstyle.xml 
> 17e55ea1071c06c73e8bf387cd69acca83bc 
>   lens-api/src/main/java/org/apache/lens/api/query/FailedAttempt.java 
> PRE-CREATION 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 
> 7d079a9ed99e3d4940d5f26370547683a6bba465 
>   lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java 
> daedbf9ea387cbde3ef39363c2beef3f96725223 
>   
> lens-api/src/test/java/org/apache/lens/api/jaxb/YAMLToStringStrategyTest.java 
> 04d9ad6dee942c705b0cdf071f49f1305696da65 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.LensQuery.xml 
> a681bca7fd6e46fe992901e04e025a8a3106 
>   
> lens-api/src/test/resources/toString/org.apache.lens.api.query.QueryStatus.xml
>  70f9c932efc090095273566eaa52d380a2f23842 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> fceabeae9217676774a6978212cbc9ee9ca120f7 
>   lens-driver-es/src/test/resources/hive-site.xml 
> e49c876289d5352203d2523ddb9134d745c197e5 
>   lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
> 84d9933da506e3f1bff2d3e0ea403ed1edf0fe35 
>   lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
> 3bf5e8f7dc9d032e46d06c3245048096acb4572e 
>   
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriverConfConstants.java
>  3c34eb191966dff3ac3032eb79282336b5c6789b 
>   
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java
>  82b56470e2de09f7c41d1226ea59cffdd8d26a26 
>   
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java
>  6e9086f9e8dc302ad2171d7f5636a24cb79dcb50 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  8cf617b7181b9a305ea2c68f68c610f823ee75d7 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/BackOffRetryHandler.java
>  17bfba02244bca7141268ed12e4a41400e2bec82 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/FailureContext.java
>  70a34b0c0517ed3b7af46120906fb77ae57870ac 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/FibonacciExponentialBackOffRetryHandler.java
>  e7fb8ce0194803317aaa0e1c5f4e3e0fb6bbca8c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/common/OperationRetryHandlerFactory.java
>  88fbe43318315394873cabb9f5dd39f09f6c47f4 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
>  03079e2dc353fa028f3c5d5a03a8ae2dbf728004 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverConfiguration.java
>  PRE-CREATION 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverQueryStatus.java
>  033f6777b4be7a1b6d6cdc350e42a94ade1acbca 
>   
> lens-server-api/src/main/java/org/apache/lens/server/