[jira] [Created] (HIVE-24342) isPathEncrypted should make sure resolved path also from HDFS
Uma Maheswara Rao G created HIVE-24342:
--
Summary: isPathEncrypted should make sure resolved path also from
HDFS
Key: HIVE-24342
URL: https://issues.apache.org/jira/browse/HIVE-24342
Project: Hive
Issue Type: Bug
Components: HiveServer2, Shims
Affects Versions: 3.1.2
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Currently isPathEncrypted will make sure path is from hdfs by check the path
scheme is "hdfs"
In the case if mounted ViewFileSystem based files systems like
ViewFSOverloadScheme or ViewHDFS (HDFS-15289) may need o check resolved path is
really hdfs.
In ViewHDFS case, we can mount hdfs://ns1/test ---> o3fs://b.v.ozone1/test
When user calling queries with the path hdfs://ns1/test, isPathEncrypted will
think the path is from hdfs only as its checking path scheme.
{code:java}
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
Error: Error while compiling statement: FAILED: SemanticException Unable to
determine if hdfs://ns1/test is encrypted:
java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
DFS. Can't run on other fs:o3fs://bucket.volume.ozone1 (state=42000,code=4)
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> cd Closing: 0:
jdbc:hive2://umag-1.umag.root.xxx.site:2181,umag-2.umag.root.xxx.site:2181,umag-5.umag.root.xxx.site:2181/default;password=root;principal=hive/umag-5.umag.root.xxx.s...@root.hwx.site;retries=5;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
{code}
So, here we should use resolvePath to make sure the resolved path really in
hdfs. If the resolved path is not from hdfs (in above case, it o3fs path), then
it will return false.
After fixing this, the query is passing.:
{code:java}
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
INFO : Compiling
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
select * from test30
INFO : No Stats for default@test30, Columns: item, user_id, state, order_id
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema:
Schema(fieldSchemas:[FieldSchema(name:test30.order_id, type:bigint,
comment:null), FieldSchema(name:test30.user_id, type:string, comment:null),
FieldSchema(name:test30.item, type:string, comment:null),
FieldSchema(name:test30.state, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb); Time
taken: 4.47 seconds
INFO : Executing
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
select * from test30
INFO : Completed executing
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb); Time
taken: 0.09 seconds
INFO : OK
+--+-+--+---+
| test30.order_id | test30.user_id | test30.item | test30.state |
+--+-+--+---+
| 1234 | u1 | iphone7 | CA |
| 2345 | u1 | ipad | CA |
| 3456 | u2 | desktop | NY |
+--+-+--+---+
11 rows selected (6.975 seconds)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-21237) [JDK 11] SessionState can't be initialized due classloader problem
Uma Maheswara Rao G created HIVE-21237:
--
Summary: [JDK 11] SessionState can't be initialized due
classloader problem
Key: HIVE-21237
URL: https://issues.apache.org/jira/browse/HIVE-21237
Project: Hive
Issue Type: Sub-task
Affects Versions: 3.1.1
Environment: JDK11, Hadoop-3, Hive 3.1.1
Reporter: Uma Maheswara Rao G
When I start Hive with JDK11
{{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
cecd9c34-d61a-44d0-9e52-a0a7d6413e49
Exception in thread "main" java.lang.ClassCastException: class
jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
java.net.URLClassLoader are in module java.base of loader 'bootstrap')
at
org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
at
org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
at
org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-2604) Add UberCompressor Serde/Codec to contrib which allows per-column compression strategies
[ https://issues.apache.org/jira/browse/HIVE-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450345#comment-13450345 ] Uma Maheswara Rao G commented on HIVE-2604: --- Hi Yongqiang, Any reason for holding this off from commit? Add UberCompressor Serde/Codec to contrib which allows per-column compression strategies Key: HIVE-2604 URL: https://issues.apache.org/jira/browse/HIVE-2604 Project: Hive Issue Type: Sub-task Components: Contrib Reporter: Krishna Kumar Assignee: Krishna Kumar Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2604.D1011.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2604.D1011.2.patch, HIVE-2604.v0.patch, HIVE-2604.v1.patch, HIVE-2604.v2.patch The strategies supported are 1. using a specified codec on the column 2. using a specific codec on the column which is serialized via a specific serde 3. using a specific TypeSpecificCompressor instance -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2623) Add Integer type compressors
[ https://issues.apache.org/jira/browse/HIVE-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433937#comment-13433937 ] Uma Maheswara Rao G commented on HIVE-2623: --- Hi Krishna, any update on your further analysis? Add Integer type compressors Key: HIVE-2623 URL: https://issues.apache.org/jira/browse/HIVE-2623 Project: Hive Issue Type: Sub-task Components: Contrib Reporter: Krishna Kumar Assignee: Krishna Kumar Priority: Minor Attachments: data.tar.gz, HIVE-2623.v0.patch, HIVE-2623.v1.patch, HIVE-2623.v2.patch Type-specific compressors for integers. Starting with elias gamma which prefers small values as per a power-law like distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-352) Make Hive support column based storage
[
https://issues.apache.org/jira/browse/HIVE-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432931#comment-13432931
]
Uma Maheswara Rao G commented on HIVE-352:
--
{quote}
trying to specify or infer best compression technique per column much harder
and something that can be done later
{quote}
Here we mentioned that, this improvement will be implemented later.
Could you please point me the right JIRA, where we started implementing this
point or discuss?
Make Hive support column based storage
--
Key: HIVE-352
URL: https://issues.apache.org/jira/browse/HIVE-352
Project: Hive
Issue Type: New Feature
Reporter: He Yongqiang
Assignee: He Yongqiang
Fix For: 0.4.0
Attachments: 4-22 performace2.txt, 4-22 performance.txt, 4-22
progress.txt, hive-352-2009-4-15.patch, hive-352-2009-4-16.patch,
hive-352-2009-4-17.patch, hive-352-2009-4-19.patch,
hive-352-2009-4-22-2.patch, hive-352-2009-4-22.patch,
hive-352-2009-4-23.patch, hive-352-2009-4-27.patch,
hive-352-2009-4-30-2.patch, hive-352-2009-4-30-3.patch,
hive-352-2009-4-30-4.patch, hive-352-2009-5-1-3.patch,
hive-352-2009-5-1.patch, HIve-352-draft-2009-03-28.patch,
Hive-352-draft-2009-03-30.patch
column based storage has been proven a better storage layout for OLAP.
Hive does a great job on raw row oriented storage. In this issue, we will
enhance hive to support column based storage.
Acctually we have done some work on column based storage on top of hdfs, i
think it will need some review and refactoring to port it to Hive.
Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
