[jira] [Commented] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647608#comment-14647608
 ] 

Rajat Khandelwal commented on HIVE-11376:
-

Taking patch from reviewboard and attaching

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11376_02.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647574#comment-14647574
 ] 

Rajat Khandelwal commented on HIVE-11376:
-

Created https://reviews.apache.org/r/36939/

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal

 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11376:

Attachment: HIVE-11376_02.patch

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11376_02.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal reassigned HIVE-11376:
---

Assignee: Rajat Khandelwal

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11376_02.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-11507) Allow Custom Authentication Provider to take HiveConf Argument

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal resolved HIVE-11507.
-
Resolution: Not A Problem

Currently custom class is instantiated with no args. The change enables 
searching a constructor with hiveconf and falls back to default constructor. So 
essentially custom provider can take a conf and configure itself.

But there’s already a way to do that, the implementation can implement 
configurable. ReflectionUtil.newInstance() takes care of rest



 Allow Custom Authentication Provider to take HiveConf Argument
 --

 Key: HIVE-11507
 URL: https://issues.apache.org/jira/browse/HIVE-11507
 Project: Hive
  Issue Type: Improvement
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11507.01.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661331#comment-14661331
 ] 

Rajat Khandelwal commented on HIVE-11376:
-

Taking patch from reviewboard and attaching

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11376.02.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11376:

Attachment: HIVE-11376.02.patch

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11376.02.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11376:

Attachment: (was: HIVE-11376_02.patch)

 CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are 
 found for one of the input files
 -

 Key: HIVE-11376
 URL: https://issues.apache.org/jira/browse/HIVE-11376
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11376.02.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
 This is the exact code snippet:
 {noformat}
 / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in 
 the tree or not,
   // we use a configuration variable for the same
   if (this.mrwork != null  !this.mrwork.getHadoopSupportsSplittable()) {
 // The following code should be removed, once
 // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
 // Hadoop does not handle non-splittable files correctly for 
 CombineFileInputFormat,
 // so don't use CombineFileInputFormat for non-splittable files
 //ie, dont't combine if inputformat is a TextInputFormat and has 
 compression turned on
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11278) Partition.setOutputFormatClass should not do toString for Class object

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11278:

Attachment: (was: HIVE-11278_2015-07-20_16:06:40.patch)

 Partition.setOutputFormatClass should not do toString for Class object 
 ---

 Key: HIVE-11278
 URL: https://issues.apache.org/jira/browse/HIVE-11278
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal

 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L286
 inside setInputFormatClass, we're doing:
 {noformat}
  public void setInputFormatClass(Class? extends InputFormat 
 inputFormatClass) {
 this.inputFormatClass = inputFormatClass;
 tPartition.getSd().setInputFormat(inputFormatClass.getName());
   }
 {noformat}
 But inside setOutputFormatClass, we're doing toString for class, instead of 
 getName().
 {noformat}
   public void setOutputFormatClass(Class? extends HiveOutputFormat 
 outputFormatClass) {
 this.outputFormatClass = outputFormatClass;
 tPartition.getSd().setOutputFormat(HiveFileFormatUtils
 .getOutputFormatSubstitute(outputFormatClass).toString());
   }
 {noformat}
 Difference is that, for Class A.class, toString is class A.class, getName 
 is A.class. So Class.forName(cls.getName()) succeeds, but 
 Class.forName(cls.toString()) is not valid. 
 So if you get a partition, set outputformat, and make an alter call, then get 
 the partition again and make a getOutputFormatClass call on that object, it 
 throws a ClassNotFoundException on 
 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L316,
  because it's basically calling Class.forName(class a.b.c.ClassName.class) 
 which is wrong!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11278) Partition.setOutputFormatClass should not do toString for Class object

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11278:

Attachment: HIVE-11278.01.patch

 Partition.setOutputFormatClass should not do toString for Class object 
 ---

 Key: HIVE-11278
 URL: https://issues.apache.org/jira/browse/HIVE-11278
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11278.01.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L286
 inside setInputFormatClass, we're doing:
 {noformat}
  public void setInputFormatClass(Class? extends InputFormat 
 inputFormatClass) {
 this.inputFormatClass = inputFormatClass;
 tPartition.getSd().setInputFormat(inputFormatClass.getName());
   }
 {noformat}
 But inside setOutputFormatClass, we're doing toString for class, instead of 
 getName().
 {noformat}
   public void setOutputFormatClass(Class? extends HiveOutputFormat 
 outputFormatClass) {
 this.outputFormatClass = outputFormatClass;
 tPartition.getSd().setOutputFormat(HiveFileFormatUtils
 .getOutputFormatSubstitute(outputFormatClass).toString());
   }
 {noformat}
 Difference is that, for Class A.class, toString is class A.class, getName 
 is A.class. So Class.forName(cls.getName()) succeeds, but 
 Class.forName(cls.toString()) is not valid. 
 So if you get a partition, set outputformat, and make an alter call, then get 
 the partition again and make a getOutputFormatClass call on that object, it 
 throws a ClassNotFoundException on 
 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L316,
  because it's basically calling Class.forName(class a.b.c.ClassName.class) 
 which is wrong!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693165#comment-14693165
 ] 

Rajat Khandelwal commented on HIVE-11483:
-

I think this should not break the existing behaviour. So a fix which decides on 
encoding depending on some properties, would be better. default behaviour can 
be to not encode. 

 Add encoding and decoding for query string config
 -

 Key: HIVE-11483
 URL: https://issues.apache.org/jira/browse/HIVE-11483
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Amareshwari Sriramadasu

 We have seen some queries in production where some of the literals passed in 
 the query have control characters, which result in exception when query 
 string is set in the job xml.
 Proposing a solution to encode the query string in configuration and provide 
 getters decoded string.
 Here is a commit in a forked repo : 
 https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
 Suggestions are welcome on the solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-11508) Allow security protocol as input in ldap authentication provider

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal resolved HIVE-11508.
-
Resolution: Not A Problem

So the changes are for getting the securityProtocol via configurations. Our use 
case was that without setting securityProtocol as 'ssl', the connection was not 
getting established. 

Later we found out that changing the ldap url to ldaps:// solves the issue. 

 Allow security protocol as input in ldap authentication provider
 

 Key: HIVE-11508
 URL: https://issues.apache.org/jira/browse/HIVE-11508
 Project: Hive
  Issue Type: Improvement
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11508.01.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11508) Allow security protocol as input in ldap authentication provider

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11508:

Attachment: HIVE-11508.01.patch

 Allow security protocol as input in ldap authentication provider
 

 Key: HIVE-11508
 URL: https://issues.apache.org/jira/browse/HIVE-11508
 Project: Hive
  Issue Type: Improvement
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11508.01.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11508) Allow security protocol as input in ldap authentication provider

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11508:

Attachment: HIVE-11508.01.patch

 Allow security protocol as input in ldap authentication provider
 

 Key: HIVE-11508
 URL: https://issues.apache.org/jira/browse/HIVE-11508
 Project: Hive
  Issue Type: Improvement
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11508.01.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11508) Allow security protocol as input in ldap authentication provider

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11508:

Attachment: (was: HIVE-11508.01.patch)

 Allow security protocol as input in ldap authentication provider
 

 Key: HIVE-11508
 URL: https://issues.apache.org/jira/browse/HIVE-11508
 Project: Hive
  Issue Type: Improvement
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11508.01.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11507) Allow Custom Authentication Provider to take HiveConf Argument

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11507:

Attachment: HIVE-11507.01.patch

 Allow Custom Authentication Provider to take HiveConf Argument
 --

 Key: HIVE-11507
 URL: https://issues.apache.org/jira/browse/HIVE-11507
 Project: Hive
  Issue Type: Improvement
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11507.01.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11278) Partition.setOutputFormatClass should not do toString for Class object

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11278:

Attachment: HIVE-11278_2015-07-20_16:06:40.patch

 Partition.setOutputFormatClass should not do toString for Class object 
 ---

 Key: HIVE-11278
 URL: https://issues.apache.org/jira/browse/HIVE-11278
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11278_2015-07-20_16:06:40.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L286
 inside setInputFormatClass, we're doing:
 {noformat}
  public void setInputFormatClass(Class? extends InputFormat 
 inputFormatClass) {
 this.inputFormatClass = inputFormatClass;
 tPartition.getSd().setInputFormat(inputFormatClass.getName());
   }
 {noformat}
 But inside setOutputFormatClass, we're doing toString for class, instead of 
 getName().
 {noformat}
   public void setOutputFormatClass(Class? extends HiveOutputFormat 
 outputFormatClass) {
 this.outputFormatClass = outputFormatClass;
 tPartition.getSd().setOutputFormat(HiveFileFormatUtils
 .getOutputFormatSubstitute(outputFormatClass).toString());
   }
 {noformat}
 Difference is that, for Class A.class, toString is class A.class, getName 
 is A.class. So Class.forName(cls.getName()) succeeds, but 
 Class.forName(cls.toString()) is not valid. 
 So if you get a partition, set outputformat, and make an alter call, then get 
 the partition again and make a getOutputFormatClass call on that object, it 
 throws a ClassNotFoundException on 
 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L316,
  because it's basically calling Class.forName(class a.b.c.ClassName.class) 
 which is wrong!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11278) Partition.setOutputFormatClass should not do toString for Class object

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633406#comment-14633406
 ] 

Rajat Khandelwal commented on HIVE-11278:
-

Small enough diff. Attaching directly

 Partition.setOutputFormatClass should not do toString for Class object 
 ---

 Key: HIVE-11278
 URL: https://issues.apache.org/jira/browse/HIVE-11278
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal
 Attachments: HIVE-11278_2015-07-20_16:06:40.patch


 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L286
 inside setInputFormatClass, we're doing:
 {noformat}
  public void setInputFormatClass(Class? extends InputFormat 
 inputFormatClass) {
 this.inputFormatClass = inputFormatClass;
 tPartition.getSd().setInputFormat(inputFormatClass.getName());
   }
 {noformat}
 But inside setOutputFormatClass, we're doing toString for class, instead of 
 getName().
 {noformat}
   public void setOutputFormatClass(Class? extends HiveOutputFormat 
 outputFormatClass) {
 this.outputFormatClass = outputFormatClass;
 tPartition.getSd().setOutputFormat(HiveFileFormatUtils
 .getOutputFormatSubstitute(outputFormatClass).toString());
   }
 {noformat}
 Difference is that, for Class A.class, toString is class A.class, getName 
 is A.class. So Class.forName(cls.getName()) succeeds, but 
 Class.forName(cls.toString()) is not valid. 
 So if you get a partition, set outputformat, and make an alter call, then get 
 the partition again and make a getOutputFormatClass call on that object, it 
 throws a ClassNotFoundException on 
 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L316,
  because it's basically calling Class.forName(class a.b.c.ClassName.class) 
 which is wrong!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083068#comment-15083068
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Created https://reviews.apache.org/r/41928/

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12776) Add code for parsing any stand-alone HQL expression

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083206#comment-15083206
 ] 

Rajat Khandelwal commented on HIVE-12776:
-

Created https://reviews.apache.org/r/41931/

> Add code for parsing any stand-alone HQL expression
> ---
>
> Key: HIVE-12776
> URL: https://issues.apache.org/jira/browse/HIVE-12776
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> Extensions that use Hive QL as their standard language, will benefit from 
> this. 
> Apache Lens uses HQL as its language of choice. To support that, it depends 
> on a fork of Hive, which has such code. I'm planning to port that to Apache 
> Hive. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.04.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12776) Add code for parsing any stand-alone HQL expression

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085103#comment-15085103
 ] 

Rajat Khandelwal commented on HIVE-12776:
-

Taking patch from reviewboard and attaching

> Add code for parsing any stand-alone HQL expression
> ---
>
> Key: HIVE-12776
> URL: https://issues.apache.org/jira/browse/HIVE-12776
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12776.01.patch, HIVE-12776.02.patch
>
>
> Extensions that use Hive QL as their standard language, will benefit from 
> this. 
> Apache Lens uses HQL as its language of choice. To support that, it depends 
> on a fork of Hive, which has such code. I'm planning to port that to Apache 
> Hive. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12776) Add code for parsing any stand-alone HQL expression

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085084#comment-15085084
 ] 

Rajat Khandelwal commented on HIVE-12776:
-

Taking patch from reviewboard and attaching

> Add code for parsing any stand-alone HQL expression
> ---
>
> Key: HIVE-12776
> URL: https://issues.apache.org/jira/browse/HIVE-12776
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12776.01.patch
>
>
> Extensions that use Hive QL as their standard language, will benefit from 
> this. 
> Apache Lens uses HQL as its language of choice. To support that, it depends 
> on a fork of Hive, which has such code. I'm planning to port that to Apache 
> Hive. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12776) Add code for parsing any stand-alone HQL expression

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12776:

Attachment: HIVE-12776.01.patch

> Add code for parsing any stand-alone HQL expression
> ---
>
> Key: HIVE-12776
> URL: https://issues.apache.org/jira/browse/HIVE-12776
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12776.01.patch
>
>
> Extensions that use Hive QL as their standard language, will benefit from 
> this. 
> Apache Lens uses HQL as its language of choice. To support that, it depends 
> on a fork of Hive, which has such code. I'm planning to port that to Apache 
> Hive. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074878#comment-15074878
 ] 

Rajat Khandelwal commented on HIVE-5672:


I see that the commit for this 
jira(https://github.com/apache/hive/commit/f3a23dce083decc565447b7438527d7aec434b29#diff-f21e965d5c189c7d150fa1e8686eb294)
 removes TOK_LOCAL_DIR altogether. Can someone elaborate why that is?

> Insert with custom separator not supported for non-local directory
> --
>
> Key: HIVE-5672
> URL: https://issues.apache.org/jira/browse/HIVE-5672
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 1.0.0
>Reporter: Romain Rigaux
>Assignee: Nemon Lou
> Fix For: 1.2.0
>
> Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch, 
> HIVE-5672.4.patch, HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz, 
> HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz, HIVE-5672.7.patch, 
> HIVE-5672.7.patch.tar.gz, HIVE-5672.8.patch, HIVE-5672.8.patch.tar.gz
>
>
> https://issues.apache.org/jira/browse/HIVE-3682 is great but non local 
> directory don't seem to be supported:
> {code}
> insert overwrite directory '/tmp/test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select description FROM sample_07
> {code}
> {code}
> Error while compiling statement: FAILED: ParseException line 2:0 cannot 
> recognize input near 'row' 'format' 'delimited' in select clause
> {code}
> This works (with 'local'):
> {code}
> insert overwrite local directory '/tmp/test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select code, description FROM sample_07
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074875#comment-15074875
 ] 

Rajat Khandelwal commented on HIVE-3682:


Please Ignore, the comment was meant to be put on HIVE-5672


> when output hive table to file,users should could have a separator of their 
> own choice
> --
>
> Key: HIVE-3682
> URL: https://issues.apache.org/jira/browse/HIVE-3682
> Project: Hive
>  Issue Type: New Feature
>  Components: CLI
>Affects Versions: 0.8.1
> Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 
> 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
> java version "1.6.0_25"
> hadoop-0.20.2-cdh3u0
> hive-0.8.1
>Reporter: caofangkun
>Assignee: Sushanth Sowmyan
> Fix For: 0.11.0
>
> Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, 
> HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, 
> HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch
>
>
> By default,when output hive table to file ,columns of the Hive table are 
> separated by ^A character (that is \001).
> But indeed users should have the right to set a seperator of their own choice.
> Usage Example:
> create table for_test (key string, value string);
> load data local inpath './in1.txt' into table for_test
> select * from for_test;
> UT-01：default separator is \001 line separator is \n
> insert overwrite local directory './test-01' 
> select * from src ;
> create table array_table (a array, b array)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ',';
> load data local inpath "../hive/examples/files/arraytest.txt" overwrite into 
> table table2;
> CREATE TABLE map_table (foo STRING , bar MAP)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ','
> MAP KEYS TERMINATED BY ':'
> STORED AS TEXTFILE;
> UT-02：defined field separator as ':'
> insert overwrite local directory './test-02' 
> row format delimited 
> FIELDS TERMINATED BY ':' 
> select * from src ;
> UT-03: line separator DO NOT ALLOWED to define as other separator 
> insert overwrite local directory './test-03' 
> row format delimited 
> FIELDS TERMINATED BY ':' 
> select * from src ;
> UT-04: define map separators 
> insert overwrite local directory './test-04' 
> row format delimited 
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ','
> MAP KEYS TERMINATED BY ':'
> select * from src;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091733#comment-15091733
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.09.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.08.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091555#comment-15091555
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093647#comment-15093647
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.11.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.12.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-13903 started by Rajat Khandelwal.
---
> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13903:

Attachment: HIVE-13903.02.patch

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch, 
> HIVE-13903.02.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324416#comment-15324416
 ] 

Rajat Khandelwal commented on HIVE-13903:
-

Taking patch from reviewboard and attaching

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch, 
> HIVE-13903.02.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313783#comment-15313783
 ] 

Rajat Khandelwal commented on HIVE-13903:
-

I don't know if there's any other elegant way that doesn't require writing 
extra code, whereas right now, we're able to accomplish this by calling an 
existing function.  Besides, the exception flow is only encountered in the 
first call -- so, still an improvement over downloading multiple times. 

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13862:

Attachment: HIVE-13862.patch

Simple enough patch. 

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13862:

Attachment: HIVE-13862.1.patch

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13862:

Attachment: (was: HIVE-13862.1.patch)

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13862:

Attachment: HIVE-13862.1.patch

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305192#comment-15305192
 ] 

Rajat Khandelwal commented on HIVE-13862:
-

Updated the patch. 

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13885) Hive session close is not resetting thread name

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13885:

Status: Patch Available  (was: Open)

> Hive session close is not resetting thread name
> ---
>
> Key: HIVE-13885
> URL: https://issues.apache.org/jira/browse/HIVE-13885
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13885.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13885) Hive session close is not resetting thread name

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13885:

Attachment: HIVE-13885.patch

[~jcamachorodriguez] This is s pretty small fix. Would like to get this too in 
2.1 release. 

> Hive session close is not resetting thread name
> ---
>
> Key: HIVE-13885
> URL: https://issues.apache.org/jira/browse/HIVE-13885
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13885.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324338#comment-15324338
 ] 

Rajat Khandelwal commented on HIVE-13903:
-

Taking patch from reviewboard and attaching

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13903:

Attachment: HIVE-13903.01.patch

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324336#comment-15324336
 ] 

Rajat Khandelwal commented on HIVE-13903:
-

Created https://reviews.apache.org/r/48544/

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13903:

Status: Patch Available  (was: In Progress)

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13885) Hive session close is not resetting thread name

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307417#comment-15307417
 ] 

Rajat Khandelwal commented on HIVE-13885:
-

Will just a logical explanation of why this change makes sense. 

HiveSessionImpl has a SessionState object (will call the instance 
sessionState). And on HiveSessionImpl.release, sessionState.resetThreadNames() 
is called, but the call is protected by null check on sessionState object. Now 
in close, we are doing sessionState.close(); sessionState = null; 
this.release;. The last call would have reset the thread name, if the 
sessionState wasn't made null just before it. Thread names are updated on 
acquire and reset on release. In close, we are doing one acquire call at the 
beginning, and one release call at the end, which means that the thread name 
will be updated but not reset (since sessionState has been made null before 
release call). 

So every thread that tries to close a session, has an extra id appended to its 
name which never gets removed. Hence the thread names keep growing and the logs 
are flooded with thread names. 

> Hive session close is not resetting thread name
> ---
>
> Key: HIVE-13885
> URL: https://issues.apache.org/jira/browse/HIVE-13885
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13885.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303586#comment-15303586
 ] 

Rajat Khandelwal commented on HIVE-13862:
-

Just to reaffirm the gravity of this fix, in our production, we had a box with 
both mysql and hive metastore running. Without this fix, both processes are 
continuously using 500-600 percent cpu each. After deploying this, the total 
cpu usage for both processes is around 50. 

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13903:

Description: on queries using permanent udfs, the jar file of the udf is 
downloaded multiple times. Each call originating from Registry.getFunctionInfo. 
This increases time for the query, especially if that query is just an explain 
query. The jar should be downloaded once, and not downloaded again if the udf 
class is accessible in the current thread. 

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310041#comment-15310041
 ] 

Rajat Khandelwal commented on HIVE-13903:
-

Created https://reviews.apache.org/r/48118/

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310042#comment-15310042
 ] 

Rajat Khandelwal commented on HIVE-13862:
-

Some issue in the pre-commit job. This issue wasn't added to pre-commit queue. 
This has happened to me before in another hive issue was working on. I've added 
manually now. 

> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
>  falls back to ORM 
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it 
> costly :
> {noformat}
>  WARN  org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed, 
> falling back to ORM
> java.lang.ClassCastException: 
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to 
> java.lang.Number
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
>  ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
>  [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-13903 started by Rajat Khandelwal.
---
> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13903:

Status: Patch Available  (was: In Progress)

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310043#comment-15310043
 ] 

Rajat Khandelwal commented on HIVE-13903:
-

Taking patch from reviewboard and attaching

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13903:

Attachment: HIVE-13903.01.patch

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097729#comment-15097729
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.13.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097581#comment-15097581
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Hi

Please take a look at the url in the description. That commit is adding restore 
functionality in cli service. In this patch I'm trying to add a bigger version 
of that. I'm adding restore functionality at the interface level. Hence, all 
the implementations have some additions. Two thrift structs have also been 
added to facilitate the addition of the functionality in one of the 
implementations. 

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.15.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105018#comment-15105018
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

This patch allows CLIService to have a functionality to restore a session given 
session handle. This functionality is added only at the service level. Clients 
don't have that functionality. Anyone that has an instance of CLIService can 
now restore previous sessions using the session handle. Have added test case 
for that too. Have updated https://reviews.apache.org/r/41928/ with the 
changes. 

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Description: 
Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Functionality added: Restoring a session. A session opened once is lost once 
the cli service is restarted. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 

Have added code in CLIService for that effect. Have also added a test class. 

  was:
Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Functionality added: Restoring a session. A session opened once is lost once 
the cli service is restarted. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 



> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105020#comment-15105020
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Description: 
Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Functionality added: Restoring a session. A session opened once is lost once 
the cli service is restarted. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 


  was:
Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.16.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105175#comment-15105175
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14113) Create function failed but function in show function list

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-14113:

Assignee: Navis  (was: niklaus xiao)

> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: Navis
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar 
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as 
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO  : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource 
> hdfs:///udf/udf-test.jar
>   at 
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
>   at 
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
>   at 
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
>   at 
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
>   at 
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
>   at 
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> hdfs:/udf/udf-test.jar
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
>   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
>   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
>   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
>   at 
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
>   ... 28 more
> ERROR : Failed to register default.my_lower using class com.tang.UDFLower
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 1 from 

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123393#comment-15123393
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Same tests failed in the build # 6787. seems to be an issue in hive master. 

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch, 
> HIVE-12777.21.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123132#comment-15123132
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch, 
> HIVE-12777.21.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.21.patch

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch, 
> HIVE-12777.21.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119443#comment-15119443
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Hi

I'd planned to rename the function, but I think renaming function makes sense 
only if there's another function that's creating a session without session 
handle. Now, there's only one function for creating a session and it allows 
null argument. 

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119603#comment-15119603
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.18.patch

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.19.patch

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120928#comment-15120928
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120927#comment-15120927
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

This is painful, master moves forward while the build is in queue. Then it's 
another day's wait before a build gets executed :/

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12972) Hive Expression parsing is significantly slow for pre-fix expression than post-fix expressions

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126030#comment-15126030
 ] 

Rajat Khandelwal commented on HIVE-12972:
-

Adding the data in table format so that it's easy to understand the difference


||Expression size||pre-fix parse time||post-fix parse time||
|0|3|2|
|1|5|3|
|2|5|3|
|3|6|11|
|4|11|7|
|5|17|6|
|6|40|6|
|7|55|10|
|8|51|5|
|9|18|6|
|10|56|4|
|11|54|5|
|12|111|5|
|13|266|9|
|14|341|4|
|15|543|4|
|16|259|2|
|17|273|2|
|18|369|3|
|19|844|2|
|20|841|3|
|21|1545|1|
|22|1882|1|
|23|2533|2|
|24|4516|2|
|25|6837|1|
|26|12679|2|
|27|21823|2|
|28|33731|1|
|29|73989|3|
|30|118986|3|

If interested, it's plotted 
[here|http://fooplot.com/#W3sidHlwZSI6MywiZXEiOltbIjAiLCIyIl0sWyIxIiwiMyJdLFsiMiIsIjMiXSxbIjMiLCIxMSJdLFsiNCIsIjciXSxbIjUiLCI2Il0sWyI2IiwiNiJdLFsiNyIsIjEwIl0sWyI4IiwiNSJdLFsiOSIsIjYiXSxbIjEwIiwiNCJdLFsiMTEiLCI1Il0sWyIxMiIsIjUiXSxbIjEzIiwiOSJdLFsiMTQiLCI0Il0sWyIxNSIsIjQiXSxbIjE2IiwiMiJdLFsiMTciLCIyIl0sWyIxOCIsIjMiXSxbIjE5IiwiMiJdLFsiMjAiLCIzIl0sWyIyMSIsIjEiXSxbIjIyIiwiMSJdLFsiMjMiLCIyIl0sWyIyNCIsIjIiXSxbIjI1IiwiMSJdLFsiMjYiLCIyIl0sWyIyNyIsIjIiXSxbIjI4IiwiMSJdLFsiMjkiLCIzIl0sWyIzMCIsIjMiXV0sImNvbG9yIjoiIzFDRkMwOCJ9LHsidHlwZSI6MywiZXEiOltbIjAiLCIzIl0sWyIxIiwiNSJdLFsiMiIsIjUiXSxbIjMiLCI2Il0sWyI0IiwiMTEiXSxbIjUiLCIxNyJdLFsiNiIsIjQwIl0sWyI3IiwiNTUiXSxbIjgiLCI1MSJdLFsiOSIsIjE4Il0sWyIxMCIsIjU2Il0sWyIxMSIsIjU0Il0sWyIxMiIsIjExMSJdLFsiMTMiLCIyNjYiXSxbIjE0IiwiMzQxIl0sWyIxNSIsIjU0MyJdLFsiMTYiLCIyNTkiXSxbIjE3IiwiMjczIl0sWyIxOCIsIjM2OSJdLFsiMTkiLCI4NDQiXSxbIjIwIiwiODQxIl0sWyIyMSIsIjE1NDUiXSxbIjIyIiwiMTg4MiJdLFsiMjMiLCIyNTMzIl0sWyIyNCIsIjQ1MTYiXSxbIjI1IiwiNjgzNyJdLFsiMjYiLCIxMjY3OSJdLFsiMjciLCIyMTgyMyJdLFsiMjgiLCIzMzczMSJdLFsiMjkiLCI3Mzk4OSJdLFsiMzAiLCIxMTg5ODYiXV0sImNvbG9yIjoiI0YyNDMxOCJ9LHsidHlwZSI6MTAwMCwid2luZG93IjpbIi01MTI5NS44NjczNDg2NjQxMSIsIjU1MjAwLjEzMjY1MTMzNTQ2NSIsIi0zMTkxMi40ODY5NzY1NTI4NDYiLCIzMzYyMy41MTMwMjM0NDY5MiJdfV0-].
 Will need to zoom-in to see the magnitude of difference. 

> Hive Expression parsing is significantly slow for pre-fix expression than 
> post-fix expressions
> --
>
> Key: HIVE-12972
> URL: https://issues.apache.org/jira/browse/HIVE-12972
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>
> Expressions
> * Prefix expression true) or true) or true) or true) takes 5 ms in 
> parsing while 
> * Postfix expression (true or (true or (true or (true takes 3 ms in 
> parsing
> Given a sequence of expressions, which are to be OR-ed, they can be OR-ed in 
> two ways. 
> Given a, b, c, d, I can make:
> * (a) -> ((a) or b) -> (((a) or b) or c) -> a) or b) or c) or d)
> * (d) -> (c or (d)) -> (b or (c or (d))) -> (a or (b or (c or (d
> The first construction is slower than the second construction, regardless of 
> the constituent sub-expressions. 
> A more important point is, as the number of sub-expressions grows, the 
> performance gap increases. Just to emphasize, it takes 2 minutes to parse a 
> pre-fix expression of 20, while it takes 3ms to parse post-fix expression of 
> 20. 
> Have tried to instrument that by writing a test case in TestParseDriver:
> I hope it's easy enough to understand:
> {noformat}
>   @Test
>   public void parseLongExpression() throws ParseException {
> List preFixParseTime = new ArrayList<>();
> List postFixParseTime = new ArrayList<>();
> String preFixExpr = "(true)";
> String postFixExpr = "(true)";
> parseDriver.parseExpression(preFixExpr);
> parseDriver.parseExpression(postFixExpr);
> for (int j = 0; j < 50; j++) {
>   preFixExpr = "(" + preFixExpr + " or true)";
>   postFixExpr = "(true or " + postFixExpr + ")";
>   long startTime = System.currentTimeMillis();
>   parseDriver.parseExpression(preFixExpr);
>   preFixParseTime.add(System.currentTimeMillis() - startTime);
>   startTime = System.currentTimeMillis();
>   parseDriver.parseExpression(postFixExpr);
>   postFixParseTime.add(System.currentTimeMillis() - startTime);
>   System.out.println("i = " + j + " prefix expr: " + preFixExpr + " 
> parseTime: " + preFixParseTime.get(j) + " postfix expr: " + postFixExpr + " 
> parse time: " + postFixParseTime.get(j));
> }
> System.out.println("done");
>   }
> {noformat}
> This was the output:
> {noformat}
> i = 0 prefix expr: ((true) or true) parseTime: 3 postfix expr: (true or 
> (true)) parse time: 2
> i = 1 prefix expr: (((true) or true) or true) parseTime: 5 postfix expr: 
> (true or (true or (true))) parse time: 3
> i = 2 prefix expr: true) or true) or true) or true) parseTime: 5 postfix 
> expr: (true or (true or (true or (true parse time: 3
> i = 3 prefix expr: (true) or true) or true) or true) or true) parseTime: 
> 6 postfix expr: (true or (true or (true or (true 

[jira] [Commented] (HIVE-12973) Create Debian in Hive packaging module

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126234#comment-15126234
 ] 

Rajat Khandelwal commented on HIVE-12973:
-

Created https://reviews.apache.org/r/43046/

> Create Debian in Hive packaging module
> --
>
> Key: HIVE-12973
> URL: https://issues.apache.org/jira/browse/HIVE-12973
> Project: Hive
>  Issue Type: New Feature
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-13179 started by Rajat Khandelwal.
---
> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171101#comment-15171101
 ] 

Rajat Khandelwal commented on HIVE-13179:
-

Created https://reviews.apache.org/r/44146/

> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13179:

Attachment: HIVE-13179.patch

> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13179.patch, HIVE-13179.patch
>
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13179:

Status: Patch Available  (was: In Progress)

Attaching patch for pre-commit job


> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13179.patch
>
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13179:

Attachment: HIVE-13179.patch

> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13179.patch
>
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13179:

Attachment: HIVE-13179.patch

> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13179.patch
>
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13179:

Attachment: (was: HIVE-13179.patch)

> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13179.patch
>
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HIVE-11483) Add encoding and decoding for query string config

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-11483 started by Rajat Khandelwal.
---
> Add encoding and decoding for query string config
> -
>
> Key: HIVE-11483
> URL: https://issues.apache.org/jira/browse/HIVE-11483
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
>
> We have seen some queries in production where some of the literals passed in 
> the query have control characters, which result in exception when query 
> string is set in the job xml.
> Proposing a solution to encode the query string in configuration and provide 
> getters decoded string.
> Here is a commit in a forked repo : 
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> Suggestions are welcome on the solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-11484) Fix ObjectInspector for Char and VarChar

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal reassigned HIVE-11484:
---

Assignee: Rajat Khandelwal  (was: Deepak Barr)

> Fix ObjectInspector for Char and VarChar
> 
>
> Key: HIVE-11484
> URL: https://issues.apache.org/jira/browse/HIVE-11484
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
>
> The creation of HiveChar and Varchar is not happening through ObjectInspector.
> Here is fix we pushed internally : 
> https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171898#comment-15171898
 ] 

Rajat Khandelwal commented on HIVE-11483:
-

Created https://reviews.apache.org/r/44172/

> Add encoding and decoding for query string config
> -
>
> Key: HIVE-11483
> URL: https://issues.apache.org/jira/browse/HIVE-11483
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
>
> We have seen some queries in production where some of the literals passed in 
> the query have control characters, which result in exception when query 
> string is set in the job xml.
> Proposing a solution to encode the query string in configuration and provide 
> getters decoded string.
> Here is a commit in a forked repo : 
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> Suggestions are welcome on the solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11483) Add encoding and decoding for query string config

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11483:

Attachment: HIVE-11483.01.patch

> Add encoding and decoding for query string config
> -
>
> Key: HIVE-11483
> URL: https://issues.apache.org/jira/browse/HIVE-11483
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Attachments: HIVE-11483.01.patch
>
>
> We have seen some queries in production where some of the literals passed in 
> the query have control characters, which result in exception when query 
> string is set in the job xml.
> Proposing a solution to encode the query string in configuration and provide 
> getters decoded string.
> Here is a commit in a forked repo : 
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> Suggestions are welcome on the solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11483) Add encoding and decoding for query string config

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-11483:

Status: Patch Available  (was: In Progress)

> Add encoding and decoding for query string config
> -
>
> Key: HIVE-11483
> URL: https://issues.apache.org/jira/browse/HIVE-11483
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Attachments: HIVE-11483.01.patch
>
>
> We have seen some queries in production where some of the literals passed in 
> the query have control characters, which result in exception when query 
> string is set in the job xml.
> Proposing a solution to encode the query string in configuration and provide 
> getters decoded string.
> Here is a commit in a forked repo : 
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> Suggestions are welcome on the solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13179:

Attachment: HIVE-13179.1.patch

Missed one line, attaching patch again.

> Allow custom HiveConf to be passed to Authentication Providers
> --
>
> Key: HIVE-13179
> URL: https://issues.apache.org/jira/browse/HIVE-13179
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13179.1.patch, HIVE-13179.patch, HIVE-13179.patch
>
>
> Right now if I want to create an ldap auth provider, I have to create a 
> hive-site.xml, set endpoints and other relevant properties there, then 
> instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
> new HiveConf is constructed. 
> A better and more reusable design would be to ask for the conf in the 
> constructor itself. That will allow an external user to create a HiveConf, 
> set all relevant properties and instantiate `LdapAuthenticationProviderImpl` 
> with that conf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110606#comment-15110606
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Yes, found something that is causing NPE. fixed and updated. 

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110590#comment-15110590
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Taking patch from reviewboard and attaching

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Attachment: HIVE-12777.17.patch

> Add capability to restore session
> -
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Description: 
Extensions using CLIService session handles should be able to restore the 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Functionality added: Restoring a session. A session opened once is lost once 
the cli service is re-created. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 

Have added code in CLIService for that effect. Have also added a test class. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be

  was:
Extensions using CLIService session handles should be able to restore the 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Functionality added: Restoring a session. A session opened once is lost once 
the cli service is re-created. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 

Have added code in CLIService for that effect. Have also added a test class. 


> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112367#comment-15112367
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

In future, this feature can be used to add session restore capabilities in Hive 
Server. If HIveServer can persist SessionHandle objects during restart, then it 
can very well restore them with some additional code. Will take that up in a 
follow up jira. 

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Description: 
Extensions using CLIService session handles should be able to restore the 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Functionality added: Restoring a session. A session opened once is lost once 
the cli service is re-created. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 

Have added code in CLIService for that effect. Have also added a test class. 

  was:
Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Functionality added: Restoring a session. A session opened once is lost once 
the cli service is restarted. There may be some operation going on in that 
session at the time the service is restarted. It's useful to be able to restore 
a previously existing session. 

Have added code in CLIService for that effect. Have also added a test class. 


> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-12777:

Summary: Add capability to restore session in CLIService and SessionManager 
 (was: Add capability to restore session)

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is restarted. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114805#comment-15114805
 ] 

Rajat Khandelwal commented on HIVE-12777:
-

Agree with you on that. Maybe the name can be changed to 
`retrieveSessionHandle` or something. However, notice that the person who's 
trying to retrieve then handle, had persisted the handle. The same person can 
persist the jars, functions etc as and when required, so that they can be used 
at the time or restore. The full-fledged restore feature that I mentioned 
earlier will contain the steps you mention. This feature is just for the users 
of CLIService. 

> Add capability to restore session in CLIService and SessionManager
> --
>
> Key: HIVE-12777
> URL: https://issues.apache.org/jira/browse/HIVE-12777
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, 
> HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, 
> HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, 
> HIVE-12777.17.patch
>
>
> Extensions using CLIService session handles should be able to restore the 
> session from the handle. 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> Functionality added: Restoring a session. A session opened once is lost once 
> the cli service is re-created. There may be some operation going on in that 
> session at the time the service is restarted. It's useful to be able to 
> restore a previously existing session. 
> Have added code in CLIService for that effect. Have also added a test class. 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)