[jira] [Created] (DRILL-6293) Unable to read hive(2.1.1) tables using Drill 1.13.0
Anup Tiwari created DRILL-6293:
--
Summary: Unable to read hive(2.1.1) tables using Drill 1.13.0
Key: DRILL-6293
URL: https://issues.apache.org/jira/browse/DRILL-6293
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Anup Tiwari
Hi,
{color:#22}I am not able to read my hive tables in drill 1.13.0 and with
same plugin conf it was working in Drill 1.12.0 and 1.10.0.{color}
*Hive Plugin :-*
{
"type": "hive",
"enabled": true,
"configProps": {
"hive.metastore.uris":
"thrift://[prod-hadoop-1xx.com:9083|http://prod-hadoop-1xx.com:9083/]";,
"hive.metastore.sasl.enabled": "false",
"[fs.default.name|http://fs.default.name/]":
"hdfs://[prod-hadoop-1xx.com:9000|http://prod-hadoop-1xx.com:9000/]";
}
}
*Query :-*
select id from hive.cad where log_date = '2018-03-18' limit 3;
*Error :-*
2018-03-20 14:25:27,351 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
254f337f-9ac3-b66f-ed17-1de459da3283: select id from hive.cad where log_date =
'2018-03-18' limit 3
2018-03-20 14:25:27,354 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] WARN
o.a.d.e.s.h.DrillHiveMetaStoreClient - Failure while attempting to get hive
table. Retries once.
org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1344)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.getHiveReadEntryHelper(DrillHiveMetaStoreClient.java:285)
~[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:535)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:531)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
[guava-18.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
[guava-18.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
[guava-18.0.jar:na]
at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getHiveReadEntry(DrillHiveMetaStoreClient.java:495)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSelectionBaseOnName(HiveSchemaFactory.java:233)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getDrillTable(HiveSchemaFactory.java:213)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.schema.HiveDatabaseSchema.getTable(HiveDatabaseSchema.java:62)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getTable(HiveSchemaFactory.java:201)
[drill-storage-hive-core-1.13.0.jar:1.13.0]
at
org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:82)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:257)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntryFrom(SqlValidatorUtil.java:1003)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntry(SqlValidatorUtil.java:960)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:117)
[calcite-core-1.15.0-drill-r0
Re: [Drill 1.13.0] : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
Note : Using Show databases, i can see hive schemas.
On Tue, Mar 20, 2018 2:36 PM, Anup Tiwari anup.tiw...@games24x7.com wrote:
Hi,
I am not able to read my hive tables in drill 1.13.0 and with same plugin conf
it was working in Drill 1.12.0 and 1.10.0. Please look into it asap and let me
know if i have missed anything.
Hive Plugin :-
{ "type": "hive", "enabled": true, "configProps": {"hive.metastore.uris":
"thrift://prod-hadoop-1xx.com:9083","hive.metastore.sasl.enabled": "false",
"fs.default.name": "hdfs://prod-hadoop-1xx.com:9000" }}
Query :-
select id from hive.cad where log_date = '2018-03-18' limit 3
Error :-
2018-03-20 14:25:27,351 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
254f337f-9ac3-b66f-ed17-1de459da3283: select id from hive.cad where log_date =
'2018-03-18' limit 32018-03-20 14:25:27,354
[254f337f-9ac3-b66f-ed17-1de459da3283:foreman] WARN
o.a.d.e.s.h.DrillHiveMetaStoreClient - Failure while attempting to get hive
table. Retries once.org.apache.thrift.TApplicationException: Invalid method
name: 'get_table_req' at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0] at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0] at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0] at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0] at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1344)
~[drill-hive-exec-shaded-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.getHiveReadEntryHelper(DrillHiveMetaStoreClient.java:285)
~[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:535)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:531)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
[guava-18.0.jar:na] at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
[guava-18.0.jar:na] at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
[guava-18.0.jar:na] at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
[guava-18.0.jar:na] at
com.google.common.cache.LocalCache.get(LocalCache.java:3937) [guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
[guava-18.0.jar:na] at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
[guava-18.0.jar:na] at
org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getHiveReadEntry(DrillHiveMetaStoreClient.java:495)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSelectionBaseOnName(HiveSchemaFactory.java:233)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getDrillTable(HiveSchemaFactory.java:213)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.schema.HiveDatabaseSchema.getTable(HiveDatabaseSchema.java:62)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getTable(HiveSchemaFactory.java:201)
[drill-storage-hive-core-1.13.0.jar:1.13.0] at
org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:82)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at
org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:257)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntryFrom(SqlValidatorUtil.java:1003)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntry(SqlValidatorUtil.java:960)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at
org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:117)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at
org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable(SqlConverter.java:633)
[drill-java-exec-1.13.0.jar:1.13.0] at
org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom(SqlConverter.java:261)
[drill-java-exec-1.13.0.jar:1.13.0] at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidator
[Drill 1.13.0] : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
SqlValidatorImpl.validate(SqlValidatorImpl.java:613) ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:190) [drill-java-exec-1.13.0.jar:1.13.0] ... 10 common frames omittedCaused by: org.apache.calcite.sql.validate.SqlValidatorException: Object 'cad' not found within 'hive' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_72] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_72] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_72] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_72] at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] ... 31 common frames omitted 2018-03-20 14:25:27,375 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] INFO o.apache.drill.exec.work.WorkManager - Waiting for 0 queries to complete before shutting down2018-03-20 14:25:27,375 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] INFO o.apache.drill.exec.work.WorkManager - Waiting for 0 running fragments to complete before shutting down Regards, Anup Tiwari
Re: [Drill 1.12.0] : Suggestions on Downgrade to 1.11.0 & com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
Hi All, We checked our MySQL max number of connections which is set to 200 and i think this might be due to exceeding max number of connections only as right now i can see 89 connections to MySQL. I want to know community's thoughts on this whether i am heading in right direction or not. On Fri, Mar 16, 2018 1:03 PM, Anup Tiwari anup.tiw...@games24x7.com wrote: Hi All, We are getting a lot of different type of issues/error post upgrading from Drill 1.10.0 to 1.12.0 which i am asking on forum as well so just wanted to know whether downgrading to Drill 1.11.0 will help or not? This time we got exception related to mysql connection storage and please note that this issue is not consistent i.e. if i execute this query after some time then it works. Please find below query are error logs. Query : create table dfs.tmp.table_info as select * from mysql.test.table_info; Error : WARN o.a.d.e.store.jdbc.JdbcStoragePlugin - Failure while attempting to load JDBC schema.com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 49,949,177 milliseconds ago. The last packet sent successfully to the server was 49,949,196 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_72]at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_72]at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_72]at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_72] at com.mysql.jdbc.Util.handleNewInstance(Util.java:389) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1038) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3609) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2417) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2531) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1446) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at com.mysql.jdbc.DatabaseMetaData.getCatalogs(DatabaseMetaData.java:2025) ~[mysql-connector-java-5.1.35-bin.jar:5.1.35]at org.apache.commons.dbcp.DelegatingDatabaseMetaData.getCatalogs(DelegatingDatabaseMetaData.java:190) ~[commons-dbcp-1.4.jar:1.4]at org.apache.drill.exec.store.jdbc.JdbcStoragePlugin$JdbcCatalogSchema.(JdbcStoragePlugin.java:309) ~[drill-jdbc-storage-1.12.0.jar:1.12.0]at org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.registerSchemas(JdbcStoragePlugin.java:430) [drill-jdbc-storage-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.DynamicRootSchema.loadSchemaFactory(DynamicRootSchema.java:94) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.DynamicRootSchema.getSubSchema(DynamicRootSchema.java:74) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.calcite.prepare.CalciteCatalogReader.getSchema(CalciteCatalogReader.java:160) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:114) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:108) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable(SqlConverter.java:493) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable(SqlConverter.java:434) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:75) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:124) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.A
[Drill 1.12.0] : Suggestions on Downgrade to 1.11.0 & com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
drill-r23]at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:886) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:872) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2817) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2802) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3025) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:886) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:872) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:210) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:846) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:560) [calcite-core-1.4.0-drill-r23.jar:1.4.0-drill-r23]at org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:172) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:617) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:192) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.handlers.CreateTableHandler.getPlan(CreateTableHandler.java:77) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1017) [drill-java-exec-1.12.0.jar:1.12.0]at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:289) [drill-java-exec-1.12.0.jar:1.12.0]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] Let me know what to do here. Regards, Anup Tiwari
Re: Does s3 plugin support AWS S3 signature version 4 ?
Any updates on this? Since we have migrated to Aws Mumbai, we are not able to connect s3 and Drill. On 04-Apr-2017 11:02 PM, "Shankar Mane" wrote: > Quick question here: > > Does s3 plugin support S3 signature version 4 ? > > FYI: s3 plugin works in case when region has support for both v2 and v4 > signature. Whereas it seems problematic, for regions (eg. ap-south-1) which > only has v4 signature version support. > > regards, > shankar >
Re: Running cartesian joins on Drill
Hi, I have one question here.. so if we have to use Cartesian join in Drill then do we have to follow some workaround like Shadi mention : adding a dummy column on the fly that has the value 1 in both tables and then join on that column leading to having a match of every row of the first table with every row of the second table, hence do a Cartesian product? OR If we just don't specify join condition like : select a.*, b.* from tt1 as a, tt2 b; then will it internally treat this query as Cartesian join. Regards, *Anup Tiwari* On Mon, May 8, 2017 at 10:00 PM, Zelaine Fong wrote: > Cartesian joins in Drill are implemented as nested loop joins, and I think > you should see that reflected in the resultant query plan when you run > explain plan on the query. > > Yes, Cartesian joins/nested loop joins are expensive because you’re > effectively doing an MxN read of your tables. There are more efficient > ways of processing a nested loop join, e.g., by creating an index on the > larger table in the join and then using that index to do lookups into that > table. That way, the nested loop join cost is the cost of creating the > index + M, where M is the number of rows in the smaller table and assuming > the lookup cost into the index does minimize the amount of data read of the > second table. Drill currently doesn’t do this. > > -- Zelaine > > On 5/8/17, 9:09 AM, "Muhammad Gelbana" wrote: > > I believe clhubert is referring to this discussion > <http://drill-user.incubator.apache.narkive.com/TIXWiTY4/ > cartesian-product-in-apache-drill#post1> > . > > So why Drill doesn't transform this query into a nested join query ? > Simply > because there is no Calcite rule to transform it into a nested loop > join ? > Is it not technically possible to write such Rule or is it feasible so > I > may take on this challenge ? > > Also pardon me for repeating my question but I fail to find an answer > in > your replies, why doesn't Drill just run a cartesian join ? Because > it's > expensive regarding resources (i.e. CPU\Network\RAM) ? > > Thanks a lot Shadi for the query, it works for me. > > *-* > *Muhammad Gelbana* > http://www.linkedin.com/in/mgelbana > > On Mon, May 8, 2017 at 6:10 AM, Shadi Khalifa > wrote: > > > Hi Muhammad, > > > > I did the following as a workaround to have Cartesian product. The > basic > > idea is to create a dummy column on the fly that has the value 1 in > both > > tables and then join on that column leading to having a match of > every row > > of the first table with every row of the second table, hence do a > Cartesian > > product. This might not be the most efficient way but it will do the > job. > > > > *Original Query:* > > SELECT * FROM > > ( SELECT 'ABC' `UserID` FROM `dfs`.`path_to_parquet_file` tc LIMIT > > 2147483647) `t0` > > INNER JOIN > > ( SELECT 'ABC' `UserID` FROM `dfs`.`path_to_parquet_file` tc LIMIT > > 2147483647) `t1` > > ON (`t0`.`UserID` IS NOT DISTINCT FROM `t1`.`UserID`) > > LIMIT 2147483647 > > > > *Workaround (add columns **d1a381f3g73 and **d1a381f3g74 to tables > one > > and two, respectively. Names don't really matter, just need to be > unique):* > > SELECT * FROM > > ( SELECT *1 as d1a381f3g73*, 'ABC' `UserID` FROM > > `dfs`.`path_to_parquet_file` tc LIMIT 2147483647) `t0` > > INNER JOIN > > ( SELECT *1 as d1a381f3g74*, 'ABC' `UserID` FROM > > `dfs`.`path_to_parquet_file` tc LIMIT 2147483647) `t1` > > ON (`t0`.*d1a381f3g73 = *`t1`.*d1a381f3g74*) > > WHERE `t0`.`UserID` IS NOT DISTINCT FROM `t1`.`UserID` > > LIMIT 2147483647 > > > > Regards > > > > > > *Shadi Khalifa, PhD* > > Postdoctoral Fellow > > Cognitive Analytics Development Hub > > Centre for Advanced Computing > > Queen’s University > > (613) 533-6000 x78347 > > http://cac.queensu.ca > > > > I'm just a neuron in the society collective brain > > > > *Join us for HPCS in June 2017! Register at:* *http://2017.hpcs.ca/ > > <http://2017.hpcs.ca/>* > > > > P Please consider your environmental responsibility before printing > this > > e-mail > > > > *01001001 0010 01101100 0110 01110110 01100101 0010 > 01000101 > > 01100111 0001 0111 01110100 * > >
Re: [Drill 1.10.0] : Memory was leaked by query
Thanks Padma, it worked. Regards, *Anup Tiwari* On Wed, Apr 19, 2017 at 1:13 AM, Kunal Khatua wrote: > Could you also share the profiles for the failed queries as well? > > > Thanks > > Kunal > > > From: Padma Penumarthy > Sent: Tuesday, April 18, 2017 7:18:08 AM > To: u...@drill.apache.org > Cc: dev@drill.apache.org > Subject: Re: [Drill 1.10.0] : Memory was leaked by query > > Seems like you are running into DRILL-5435<https://issues.apac > he.org/jira/browse/DRILL-5435>. > Try turning off async parquet reader and see if that helps. > alter session set `store.parquet.reader.pagereader.async`=false; > > Thanks, > Padma > > > On Apr 18, 2017, at 6:14 AM, Anup Tiwari lto:anup.tiw...@games24x7.com>> wrote: > > Hi Team, > > Please find following information : > > *Cluster configuration :* > Number of Nodes : 5 > Cores/Node : 8 > RAM : 32 > > *Variable values :* > planner.width.max_per_node = 5 > planner.width.max_per_query = 30 > planner.memory.max_query_memory_per_node = 4294967296 > > I am getting following error on simple select statement which is coming 6 > times out of 10 times, let me know if i am missing anything: > > *Query :* > select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl > like '%/j_check%' and ajaxResponse like '%success%true%') limit 5; > > *Error :* > > ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) > Allocator(op:1:24:6:ParquetRowGroupScan) > 100/1048576/27140096/100 (res/actual/peak/limit) > > > Fragment 1:24 > > [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on > prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo > p-101.bom-prod.aws.games24x7.com>:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (1048576) > Allocator(op:1:24:6:ParquetRowGroupScan) > 100/1048576/27140096/100 (res/actual/peak/limit) > > > Fragment 1:24 > > [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on > prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo > p-101.bom-prod.aws.games24x7.com>:31010] >at > org.apache.drill.common.exceptions.UserException$Builder. > build(UserException.java:544) > ~[drill-common-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin > alState(FragmentExecutor.java:293) > [drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup > (FragmentExecutor.java:160) > [drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra > gmentExecutor.java:262) > [drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin > gRunnable.java:38) > [drill-common-1.10.0.jar:1.10.0] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1142) > [na:1.8.0_72] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:617) > [na:1.8.0_72] >at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: java.lang.IllegalStateException: Memory was leaked by query. > Memory leaked: (1048576) > Allocator(op:1:24:6:ParquetRowGroupScan) > 100/1048576/27140096/100 (res/actual/peak/limit) > >at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502) > ~[drill-memory-base-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.ops.OperatorContextImpl.close(Operator > ContextImpl.java:149) > ~[drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(F > ragmentContext.java:422) > ~[drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411) > ~[drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu > tResources(FragmentExecutor.java:318) > [drill-java-exec-1.10.0.jar:1.10.0] >at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup > (FragmentExecutor.java:155) > [drill-java-exec-1.10.0.jar:1.10.0] >... 5 common frames omitted > 2017-04-18 18:21:54,172 [BitServer-4] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING > --> CANCELLATION_REQUESTED > 2017-04-18 18:21:54,172 [BitServer-4] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: Sta
[Drill 1.10.0] : Memory was leaked by query
00 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
100/1048576/27140096/100 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*
Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?
Hi John, First of all sorry for delayed response and thanks for your suggestion, reducing value of "planner.width.max_per_node" helped me a lot, above issue which was coming 8 out of 10 times earlier now it is coming only 2 out of 10 times. As mentioned above occurrences of connection error came down considerably, but now sometimes i get "Heap Space Error" for few queries and due to this sometimes drill-bits on some/all nodes gets killed. Let me know if any other variable i can check for this(As of now, i have 8GB of Heap and 20GB of Direct memory) : *Error Log :* ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. Information message: Unable to handle out of memory condition in FragmentExecutor. java.lang.OutOfMemoryError: Java heap space at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(Unknown Source) ~[xercesImpl-2.11.0.jar:na] at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknown Source) ~[xercesImpl-2.11.0.jar:na] at org.apache.xerces.dom.DeferredElementImpl.synchronizeChildren(Unknown Source) ~[xercesImpl-2.11.0.jar:na] at org.apache.xerces.dom.ElementImpl.normalize(Unknown Source) ~[xercesImpl-2.11.0.jar:na] at org.apache.xerces.dom.ElementImpl.normalize(Unknown Source) ~[xercesImpl-2.11.0.jar:na] at org.apache.xerces.dom.ElementImpl.normalize(Unknown Source) ~[xercesImpl-2.11.0.jar:na] at com.games24x7.device.NewDeviceData.setup(NewDeviceData.java:94) ~[DeviceDataClient-0.0.1-SNAPSHOT.jar:na] at org.apache.drill.exec.test.generated.FiltererGen5369.doSetup(FilterTemplate2.java:97) ~[na:na] at org.apache.drill.exec.test.generated.FiltererGen5369.setup(FilterTemplate2.java:54) ~[na:na] at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer(FilterRecordBatch.java:195) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema(FilterRecordBatch.java:107) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.buildSchema(HashAggBatch.java:108) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] Regards, *Anup Tiwari* On Mon, Mar 6, 2017 at 7
Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?
Hi John, I have tried above config as well but still getting this issue. And please note that we were using similar configuration params for Drill 1.6 where this issue was not coming. Anything else which i can try? Regards, *Anup Tiwari* On Fri, Mar 3, 2017 at 11:01 PM, Abhishek Girish wrote: > +1 on John's suggestion. > > On Fri, Mar 3, 2017 at 6:24 AM, John Omernik wrote: > > > So your node has 32G of ram yet you are allowing Drill to use 36G. I > would > > change your settings to be 8GB of Heap, and 22GB of Direct Memory. See if > > this helps with your issues. Also, are you using a distributed > filesystem? > > If so you may want to allow even more free ram...i.e. 8GB of Heap and > 20GB > > of Direct. > > > > On Fri, Mar 3, 2017 at 8:20 AM, Anup Tiwari > > wrote: > > > > > Hi, > > > > > > Please find our configuration details :- > > > > > > Number of Nodes : 4 > > > RAM/Node : 32GB > > > Core/Node : 8 > > > DRILL_MAX_DIRECT_MEMORY="20G" > > > DRILL_HEAP="16G" > > > > > > And all other variables are set to default. > > > > > > Since we have tried some of the settings suggested above but still > facing > > > this issue more frequently, kindly suggest us what is best > configuration > > > for our environment. > > > > > > Regards, > > > *Anup Tiwari* > > > > > > On Thu, Mar 2, 2017 at 1:26 AM, John Omernik wrote: > > > > > > > Another thing to consider is ensure you have a Spill Location setup, > > and > > > > then disable hashagg/hashjoin for the query... > > > > > > > > On Wed, Mar 1, 2017 at 1:25 PM, Abhishek Girish > > > > wrote: > > > > > > > > > Hey Anup, > > > > > > > > > > This is indeed an issue, and I can understand that having an > unstable > > > > > environment is not something anyone wants. DRILL-4708 is still > > > > unresolved - > > > > > hopefully someone will get to it soon. I've bumped up the priority. > > > > > > > > > > Unfortunately we do not publish any sizing guidelines, so you'd > have > > to > > > > > experiment to settle on the right load for your cluster. Please > > > decrease > > > > > the concurrency (number of queries running in parallel). And try > > > bumping > > > > up > > > > > Drill DIRECT memory. Also, please set the system options > recommended > > by > > > > > Sudheesh. While this may not solve the issue, it may help reduce > it's > > > > > occurrence. > > > > > > > > > > Can you also update the JIRA with your configurations, type of > > queries > > > > and > > > > > the relevant logs? > > > > > > > > > > -Abhishek > > > > > > > > > > On Wed, Mar 1, 2017 at 10:17 AM, Anup Tiwari < > > > anup.tiw...@games24x7.com> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Can someone look into it? As we are now getting this more > > frequently > > > in > > > > > > Adhoc queries as well. > > > > > > And for automation jobs, we are moving to Hive as in drill we are > > > > getting > > > > > > this more frequently. > > > > > > > > > > > > Regards, > > > > > > *Anup Tiwari* > > > > > > > > > > > > On Sat, Dec 31, 2016 at 12:11 PM, Anup Tiwari < > > > > anup.tiw...@games24x7.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > We are getting this issue bit more frequently. can someone > please > > > > look > > > > > > > into it and tell us that why it is happening since as mention > in > > > > > earlier > > > > > > > mail when this query gets executed no other query is running at > > > that > > > > > > time. > > > > > > > > > > > > > > Thanks in advance. > > > > > > > > > > > > > > Regards, > > > > > > > *Anup Tiwari* > > > > > > > > &g
Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?
Hi, Please find our configuration details :- Number of Nodes : 4 RAM/Node : 32GB Core/Node : 8 DRILL_MAX_DIRECT_MEMORY="20G" DRILL_HEAP="16G" And all other variables are set to default. Since we have tried some of the settings suggested above but still facing this issue more frequently, kindly suggest us what is best configuration for our environment. Regards, *Anup Tiwari* On Thu, Mar 2, 2017 at 1:26 AM, John Omernik wrote: > Another thing to consider is ensure you have a Spill Location setup, and > then disable hashagg/hashjoin for the query... > > On Wed, Mar 1, 2017 at 1:25 PM, Abhishek Girish > wrote: > > > Hey Anup, > > > > This is indeed an issue, and I can understand that having an unstable > > environment is not something anyone wants. DRILL-4708 is still > unresolved - > > hopefully someone will get to it soon. I've bumped up the priority. > > > > Unfortunately we do not publish any sizing guidelines, so you'd have to > > experiment to settle on the right load for your cluster. Please decrease > > the concurrency (number of queries running in parallel). And try bumping > up > > Drill DIRECT memory. Also, please set the system options recommended by > > Sudheesh. While this may not solve the issue, it may help reduce it's > > occurrence. > > > > Can you also update the JIRA with your configurations, type of queries > and > > the relevant logs? > > > > -Abhishek > > > > On Wed, Mar 1, 2017 at 10:17 AM, Anup Tiwari > > wrote: > > > > > Hi, > > > > > > Can someone look into it? As we are now getting this more frequently in > > > Adhoc queries as well. > > > And for automation jobs, we are moving to Hive as in drill we are > getting > > > this more frequently. > > > > > > Regards, > > > *Anup Tiwari* > > > > > > On Sat, Dec 31, 2016 at 12:11 PM, Anup Tiwari < > anup.tiw...@games24x7.com > > > > > > wrote: > > > > > > > Hi, > > > > > > > > We are getting this issue bit more frequently. can someone please > look > > > > into it and tell us that why it is happening since as mention in > > earlier > > > > mail when this query gets executed no other query is running at that > > > time. > > > > > > > > Thanks in advance. > > > > > > > > Regards, > > > > *Anup Tiwari* > > > > > > > > On Sat, Dec 24, 2016 at 10:20 AM, Anup Tiwari < > > anup.tiw...@games24x7.com > > > > > > > > wrote: > > > > > > > >> Hi Sudheesh, > > > >> > > > >> Please find below ans :- > > > >> > > > >> 1. Total 4,(3 Datanodes, 1 namenode) > > > >> 2. Only one query, as this query is part of daily dump and runs in > > early > > > >> morning. > > > >> > > > >> And as @chun mentioned , it seems similar to DRILL-4708 , so any > > update > > > >> on progress of this ticket? > > > >> > > > >> > > > >> On 22-Dec-2016 12:13 AM, "Sudheesh Katkam" > > > wrote: > > > >> > > > >> Two more questions.. > > > >> > > > >> (1) How many nodes in your cluster? > > > >> (2) How many queries are running when the failure is seen? > > > >> > > > >> If you have multiple large queries running at the same time, the > load > > on > > > >> the system could cause those failures (which are heartbeat related). > > > >> > > > >> The two options I suggested decrease the parallelism of stages in a > > > >> query, this implies lesser load but slower execution. > > > >> > > > >> System level option affect all queries, and session level affect > > queries > > > >> on a specific connection. Not sure what is preferred in your > > > environment. > > > >> > > > >> Also, you may be interested in metrics. More info here: > > > >> > > > >> http://drill.apache.org/docs/monitoring-metrics/ < > > > >> http://drill.apache.org/docs/monitoring-metrics/> > > > >> > > > >> Thank you, > > > >> Sudheesh > > > >> > > > >> > On Dec 21, 2016, at 4:31 AM, Anup Tiwari < > anup.tiw...@games24x7.com > > > > > > >>
Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?
Hi, Can someone look into it? As we are now getting this more frequently in Adhoc queries as well. And for automation jobs, we are moving to Hive as in drill we are getting this more frequently. Regards, *Anup Tiwari* On Sat, Dec 31, 2016 at 12:11 PM, Anup Tiwari wrote: > Hi, > > We are getting this issue bit more frequently. can someone please look > into it and tell us that why it is happening since as mention in earlier > mail when this query gets executed no other query is running at that time. > > Thanks in advance. > > Regards, > *Anup Tiwari* > > On Sat, Dec 24, 2016 at 10:20 AM, Anup Tiwari > wrote: > >> Hi Sudheesh, >> >> Please find below ans :- >> >> 1. Total 4,(3 Datanodes, 1 namenode) >> 2. Only one query, as this query is part of daily dump and runs in early >> morning. >> >> And as @chun mentioned , it seems similar to DRILL-4708 , so any update >> on progress of this ticket? >> >> >> On 22-Dec-2016 12:13 AM, "Sudheesh Katkam" wrote: >> >> Two more questions.. >> >> (1) How many nodes in your cluster? >> (2) How many queries are running when the failure is seen? >> >> If you have multiple large queries running at the same time, the load on >> the system could cause those failures (which are heartbeat related). >> >> The two options I suggested decrease the parallelism of stages in a >> query, this implies lesser load but slower execution. >> >> System level option affect all queries, and session level affect queries >> on a specific connection. Not sure what is preferred in your environment. >> >> Also, you may be interested in metrics. More info here: >> >> http://drill.apache.org/docs/monitoring-metrics/ < >> http://drill.apache.org/docs/monitoring-metrics/> >> >> Thank you, >> Sudheesh >> >> > On Dec 21, 2016, at 4:31 AM, Anup Tiwari >> wrote: >> > >> > @sudheesh, yes drill bit is running on datanodeN/10.*.*.5:31010). >> > >> > Can you tell me how this will impact to query and do i have to set this >> at >> > session level OR system level? >> > >> > >> > >> > Regards, >> > *Anup Tiwari* >> > >> > On Tue, Dec 20, 2016 at 11:59 PM, Chun Chang >> wrote: >> > >> >> I am pretty sure this is the same as DRILL-4708. >> >> >> >> On Tue, Dec 20, 2016 at 10:27 AM, Sudheesh Katkam < >> skat...@maprtech.com> >> >> wrote: >> >> >> >>> Is the drillbit service (running on datanodeN/10.*.*.5:31010) actually >> >>> down when the error is seen? >> >>> >> >>> If not, try lowering parallelism using these two session options, >> before >> >>> running the queries: >> >>> >> >>> planner.width.max_per_node (decrease this) >> >>> planner.slice_target (increase this) >> >>> >> >>> Thank you, >> >>> Sudheesh >> >>> >> >>>> On Dec 20, 2016, at 12:28 AM, Anup Tiwari > > >> >>> wrote: >> >>>> >> >>>> Hi Team, >> >>>> >> >>>> We are running some drill automation script on a daily basis and we >> >> often >> >>>> see that some query gets failed frequently by giving below error , >> >> Also i >> >>>> came across DRILL-4708 <https://issues.apache.org/ >> >> jira/browse/DRILL-4708 >> >>>> >> >>>> which seems similar, Can anyone give me update on that OR workaround >> to >> >>>> avoid such issue ? >> >>>> >> >>>> *Stack Trace :-* >> >>>> >> >>>> Error: CONNECTION ERROR: Connection /10.*.*.1:41613 <--> >> >>>> datanodeN/10.*.*.5:31010 (user client) closed unexpectedly. Drillbit >> >>> down? >> >>>> >> >>>> >> >>>> [Error Id: 5089f2f1-0dfd-40f8-9fa0-8276c08be53f ] (state=,code=0) >> >>>> java.sql.SQLException: CONNECTION ERROR: Connection /10.*.*.1:41613 >> >> <--> >> >>>> datanodeN/10.*.*.5:31010 (user client) closed unexpectedly. Drillb >> >>>> it down? >> >>>> >> >>>> >> >>>> [Error Id: 5089f2f1-0dfd-40f8-9fa0-8276c08be53f ] >> >>>> at >> >>>> org.apache.drill.jdbc.impl.DrillCur
Re: Storage Plugin for accessing Hive ORC Table from Drill
First of all, sorry for late reply.
@Chunhui, you are right. we are using Hive 2.0
And are we planning to update hive libraries in next release of drill?
@rahul, as you said i have created another table with just "use stored as
orc" but all column and now drill is able to read it. Do you have any idea
why it worked now? Below is create table statement of new table, the
difference which i can observe is in TBLPROPERTIES ,partitioning and
bucketing:-
CREATE TABLE `logindetails_all_tmp`(
`sid` char(40),
`channel_id` tinyint,
`c_t` bigint,
`l_t` bigint)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'hdfs://hostname1:9000/usr/hive/warehouse/logindetails_all_tmp'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
'numFiles'='3',
'numRows'='1993254',
'rawDataSize'='1143386232',
'totalSize'='69876827',
'transient_lastDdlTime'='1486640969');
Regards,
*Anup Tiwari*
On Sat, Jan 21, 2017 at 1:04 PM, Chunhui Shi wrote:
> I guess you are using Hive 2.0 as meta server while Drill has only 1.2
> libraries.
>
>
> In Hive 2.0 above, This delta format could have more than one '_' as
> separator while 1.2 has only one '_'.
>
>
> I think Drill should eventually update to use Hive's 2.0/2.1 libraries.
>
>
> From: Anup Tiwari
> Sent: Friday, January 20, 2017 10:07:50 PM
> To: u...@drill.apache.org; dev@drill.apache.org
> Subject: Re: Storage Plugin for accessing Hive ORC Table from Drill
>
> @Andries, We are using Hive 2.1.1 with Drill 1.9.0.
>
> @Zelaine, Could this be a problem in your Hive metastore?--> As i mentioned
> earlier, i am able to read hive parquet tables in Drill through hive
> storage plugin. So can you tell me a bit more like which type of
> configuration i am missing in metastore?
>
> Regards,
> *Anup Tiwari*
>
> On Sat, Jan 21, 2017 at 4:56 AM, Zelaine Fong wrote:
>
> > The stack trace shows the following:
> >
> > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
> > java.io.IOException: Failed to get numRows from HiveTable
> >
> > The Drill optimizer is trying to read rowcount information from Hive.
> > Could this be a problem in your Hive metastore?
> >
> > Has anyone else seen this before?
> >
> > -- Zelaine
> >
> > On 1/20/17, 7:35 AM, "Andries Engelbrecht"
> wrote:
> >
> > What version of Hive are you using?
> >
> >
> > --Andries
> >
> >
> > From: Anup Tiwari
> > Sent: Friday, January 20, 2017 3:00:43 AM
> > To: u...@drill.apache.org; dev@drill.apache.org
> > Subject: Re: Storage Plugin for accessing Hive ORC Table from Drill
> >
> > Hi,
> >
> > Please find below Create Table Statement and subsequent Drill Error
> :-
> >
> > *Table Structure :*
> >
> > CREATE TABLE `logindetails_all`(
> > `sid` char(40),
> > `channel_id` tinyint,
> > `c_t` bigint,
> > `l_t` bigint)
> > PARTITIONED BY (
> > `login_date` char(10))
> > CLUSTERED BY (
> > channel_id)
> > INTO 9 BUCKETS
> > ROW FORMAT SERDE
> > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> > STORED AS INPUTFORMAT
> > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> > OUTPUTFORMAT
> > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> > LOCATION
> > 'hdfs://hostname1:9000/usr/hive/warehouse/logindetails_all'
> > TBLPROPERTIES (
> > 'compactorthreshold.hive.compactor.delta.num.threshold'='6',
> > 'compactorthreshold.hive.compactor.delta.pct.threshold'='0.5',
> > 'transactional'='true',
> > 'transient_lastDdlTime'='1484313383');
> > ;
> >
> > *Drill Error :*
> >
> > *Query* : select * from hive.logindetails_all limit 1;
> >
> > *Error :*
> > 2017-01-20 16:21:12,625 [277e145e-c6bc-3372-01d0-
> 6c5b75b92d73:foreman]
> > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id
> > 277e145e-c6bc-3372-01d0-6c5b75b92d73
Re: Storage Plugin for accessing Hive ORC Table from Drill
can you point me to any specific line or sentence on that link? Also please correct me if i am misinterpreting, but as written in 1st line "*Drill 1.1 and later supports Hive 1.0*", does that mean Drill 1.1 and later doesn't support OR partially support Hive 2.x? Regards, *Anup Tiwari* On Sat, Jan 21, 2017 at 8:48 PM, Zelaine Fong wrote: > Have you taken a look at http://drill.apache.org/docs/hive-storage-plugin/ > ? > > -- Zelaine > > On 1/20/17, 10:07 PM, "Anup Tiwari" wrote: > > @Andries, We are using Hive 2.1.1 with Drill 1.9.0. > > @Zelaine, Could this be a problem in your Hive metastore?--> As i > mentioned > earlier, i am able to read hive parquet tables in Drill through hive > storage plugin. So can you tell me a bit more like which type of > configuration i am missing in metastore? > > Regards, > *Anup Tiwari* > > On Sat, Jan 21, 2017 at 4:56 AM, Zelaine Fong wrote: > > > The stack trace shows the following: > > > > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: > > java.io.IOException: Failed to get numRows from HiveTable > > > > The Drill optimizer is trying to read rowcount information from Hive. > > Could this be a problem in your Hive metastore? > > > > Has anyone else seen this before? > > > > -- Zelaine > > > > On 1/20/17, 7:35 AM, "Andries Engelbrecht" > wrote: > > > > What version of Hive are you using? > > > > > > --Andries > > > > > > From: Anup Tiwari > > Sent: Friday, January 20, 2017 3:00:43 AM > > To: u...@drill.apache.org; dev@drill.apache.org > > Subject: Re: Storage Plugin for accessing Hive ORC Table from > Drill > > > > Hi, > > > > Please find below Create Table Statement and subsequent Drill > Error :- > > > > *Table Structure :* > > > > CREATE TABLE `logindetails_all`( > > `sid` char(40), > > `channel_id` tinyint, > > `c_t` bigint, > > `l_t` bigint) > > PARTITIONED BY ( > > `login_date` char(10)) > > CLUSTERED BY ( > > channel_id) > > INTO 9 BUCKETS > > ROW FORMAT SERDE > > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > > STORED AS INPUTFORMAT > > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > > OUTPUTFORMAT > > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > > LOCATION > > 'hdfs://hostname1:9000/usr/hive/warehouse/logindetails_all' > > TBLPROPERTIES ( > > 'compactorthreshold.hive.compactor.delta.num.threshold'='6', > > 'compactorthreshold.hive.compactor.delta.pct.threshold'='0.5', > > 'transactional'='true', > > 'transient_lastDdlTime'='1484313383'); > > ; > > > > *Drill Error :* > > > > *Query* : select * from hive.logindetails_all limit 1; > > > > *Error :* > > 2017-01-20 16:21:12,625 [277e145e-c6bc-3372-01d0- > 6c5b75b92d73:foreman] > > INFO o.a.drill.exec.work.foreman.Foreman - Query text for > query id > > 277e145e-c6bc-3372-01d0-6c5b75b92d73: select * from > > hive.logindetails_all > > limit 1 > > 2017-01-20 16:21:12,831 [277e145e-c6bc-3372-01d0- > 6c5b75b92d73:foreman] > > ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: > > NumberFormatException: For input string: "004_" > > > > > > [Error Id: 53fa92e1-477e-45d2-b6f7-6eab9ef1da35 on > > prod-hadoop-101.bom-prod.aws.games24x7.com:31010] > > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > > NumberFormatException: For input string: "004_" > > > > > > [Error Id: 53fa92e1-477e-45d2-b6f7-6eab9ef1da35 on > > prod-hadoop-101.bom-prod.aws.games24x7.com:31010] > > at > > org.apache.drill.common.exceptions.UserException$ > > Builder.build(UserException.java:543) > > ~[drill-common-1.9.0.jar:1.9.0] > > at > > org.apache.drill.exec.work.fo
Re: Storage Plugin for accessing Hive ORC Table from Drill
@Andries, We are using Hive 2.1.1 with Drill 1.9.0. @Zelaine, Could this be a problem in your Hive metastore?--> As i mentioned earlier, i am able to read hive parquet tables in Drill through hive storage plugin. So can you tell me a bit more like which type of configuration i am missing in metastore? Regards, *Anup Tiwari* On Sat, Jan 21, 2017 at 4:56 AM, Zelaine Fong wrote: > The stack trace shows the following: > > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: > java.io.IOException: Failed to get numRows from HiveTable > > The Drill optimizer is trying to read rowcount information from Hive. > Could this be a problem in your Hive metastore? > > Has anyone else seen this before? > > -- Zelaine > > On 1/20/17, 7:35 AM, "Andries Engelbrecht" wrote: > > What version of Hive are you using? > > > --Andries > > > From: Anup Tiwari > Sent: Friday, January 20, 2017 3:00:43 AM > To: u...@drill.apache.org; dev@drill.apache.org > Subject: Re: Storage Plugin for accessing Hive ORC Table from Drill > > Hi, > > Please find below Create Table Statement and subsequent Drill Error :- > > *Table Structure :* > > CREATE TABLE `logindetails_all`( > `sid` char(40), > `channel_id` tinyint, > `c_t` bigint, > `l_t` bigint) > PARTITIONED BY ( > `login_date` char(10)) > CLUSTERED BY ( > channel_id) > INTO 9 BUCKETS > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > LOCATION > 'hdfs://hostname1:9000/usr/hive/warehouse/logindetails_all' > TBLPROPERTIES ( > 'compactorthreshold.hive.compactor.delta.num.threshold'='6', > 'compactorthreshold.hive.compactor.delta.pct.threshold'='0.5', > 'transactional'='true', > 'transient_lastDdlTime'='1484313383'); > ; > > *Drill Error :* > > *Query* : select * from hive.logindetails_all limit 1; > > *Error :* > 2017-01-20 16:21:12,625 [277e145e-c6bc-3372-01d0-6c5b75b92d73:foreman] > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id > 277e145e-c6bc-3372-01d0-6c5b75b92d73: select * from > hive.logindetails_all > limit 1 > 2017-01-20 16:21:12,831 [277e145e-c6bc-3372-01d0-6c5b75b92d73:foreman] > ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: > NumberFormatException: For input string: "004_" > > > [Error Id: 53fa92e1-477e-45d2-b6f7-6eab9ef1da35 on > prod-hadoop-101.bom-prod.aws.games24x7.com:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > NumberFormatException: For input string: "004_" > > > [Error Id: 53fa92e1-477e-45d2-b6f7-6eab9ef1da35 on > prod-hadoop-101.bom-prod.aws.games24x7.com:31010] > at > org.apache.drill.common.exceptions.UserException$ > Builder.build(UserException.java:543) > ~[drill-common-1.9.0.jar:1.9.0] > at > org.apache.drill.exec.work.foreman.Foreman$ForemanResult. > close(Foreman.java:825) > [drill-java-exec-1.9.0.jar:1.9.0] > at > org.apache.drill.exec.work.foreman.Foreman.moveToState( > Foreman.java:935) > [drill-java-exec-1.9.0.jar:1.9.0] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman. > java:281) > [drill-java-exec-1.9.0.jar:1.9.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > [na:1.8.0_72] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > [na:1.8.0_72] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.drill.exec.work.foreman.ForemanException: > Unexpected > exception during fragment initialization: Internal error: Error while > applying rule DrillPushProjIntoScan, args > [rel#4220197:LogicalProject.NONE.ANY([]).[](input=rel# > 4220196:Subset#0.ENUMERABLE.ANY([]).[],sid=$0,channel_id=$ > 1,c_t=$2,l_t=$3,login_date=$4), > rel#4220181:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[hive, > logindetails_all])] > ... 4 common frames omitted > Caused by: java.lang.AssertionError: Internal error: Error while > applying > rule DrillPushProjIntoScan, args > [rel#4220197:LogicalProject.NONE.ANY([]).[](in
Re: Storage Plugin for accessing Hive ORC Table from Drill
iveCost(RelMdPercentageOriginalRows.java:165) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] ... 42 common frames omitted Caused by: java.io.IOException: Failed to get numRows from HiveTable at org.apache.drill.exec.store.hive.HiveMetadataProvider.getStats(HiveMetadataProvider.java:113) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] at org.apache.drill.exec.store.hive.HiveScan.getScanStats(HiveScan.java:224) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] ... 45 common frames omitted Caused by: java.lang.RuntimeException: serious problem at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] at org.apache.drill.exec.store.hive.HiveMetadataProvider$1.run(HiveMetadataProvider.java:253) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] at org.apache.drill.exec.store.hive.HiveMetadataProvider$1.run(HiveMetadataProvider.java:241) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_72] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_72] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[hadoop-common-2.7.1.jar:na] at org.apache.drill.exec.store.hive.HiveMetadataProvider.splitInputWithUGI(HiveMetadataProvider.java:241) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] at org.apache.drill.exec.store.hive.HiveMetadataProvider.getPartitionInputSplits(HiveMetadataProvider.java:142) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] at org.apache.drill.exec.store.hive.HiveMetadataProvider.getStats(HiveMetadataProvider.java:105) ~[drill-storage-hive-core-1.9.0.jar:1.9.0] ... 46 common frames omitted Caused by: java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: "004_" at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_72] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_72] at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:998) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] ... 55 common frames omitted Caused by: java.lang.NumberFormatException: For input string: "004_" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.8.0_72] at java.lang.Long.parseLong(Long.java:589) ~[na:1.8.0_72] at java.lang.Long.parseLong(Long.java:631) ~[na:1.8.0_72] at org.apache.hadoop.hive.ql.io.AcidUtils.parseDelta(AcidUtils.java:310) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:379) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:634) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:620) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_72] ... 3 common frames omitted Regards, *Anup Tiwari* On Thu, Jan 19, 2017 at 9:18 PM, Andries Engelbrecht wrote: > I have not seen issues reading Hive ORC data with Drill. > > > What is the DDL for the table in Hive? > > > --Andries > > > From: Anup Tiwari > Sent: Thursday, January 19, 2017 12:49:20 AM > To: u...@drill.apache.org > Cc: dev@drill.apache.org > Subject: Re: Storage Plugin for accessing Hive ORC Table from Drill > > We have created a ORC format table in hive and we were trying to read it in > drill through hive plugin, but it is giving us error. But with same hive > plugin, we are able to read parquet table created in hive. > > So after searching a bit, i found a drill documentation link > <https://drill.apache.org/docs/apache-drill-contribution-ideas/> which > says > that we have to create custom storage plugin to read ORC format tables. So > can you tell me how to create custom storage plugin in this case? > > > > Regards, > *Anup Tiwari* > > On Thu, Jan 19, 2017 at 1:55 PM, Nitin Pawar > wrote: > > > you want to use the ORC files created by hive directly in drill or you > want > > to use them through hive? > > > > On Thu, Jan 19, 2017 at 1:40 PM, Anup Tiwari > > wrote: > > > > > +Dev > > > > > > Can someone help me in this? > > > > > > Regards, > > > *Anup Tiwari* > > > > > > On Sun, Jan 15, 2017 at 2:21 PM, Anup Tiwari < > anup.tiw...@games24x7.com> > > > wrote: > > > > > > > Hi Team, > > > > > > > > Can someone tell me how to configure custom storage plugin in Drill > for > > > > accessing hive ORC tables? > > > > > > > > Thanks in advance!! > > > > > > > > Regards, > > > > *Anup Tiwari* > > > > > > > > > > > > > > > -- > > Nitin Pawar > > >
Re: Storage Plugin for accessing Hive ORC Table from Drill
+Dev Can someone help me in this? Regards, *Anup Tiwari* On Sun, Jan 15, 2017 at 2:21 PM, Anup Tiwari wrote: > Hi Team, > > Can someone tell me how to configure custom storage plugin in Drill for > accessing hive ORC tables? > > Thanks in advance!! > > Regards, > *Anup Tiwari* >
Re: Storage Plugin for accessing Hive ORC Table from Drill
We have created a ORC format table in hive and we were trying to read it in drill through hive plugin, but it is giving us error. But with same hive plugin, we are able to read parquet table created in hive. So after searching a bit, i found a drill documentation link <https://drill.apache.org/docs/apache-drill-contribution-ideas/> which says that we have to create custom storage plugin to read ORC format tables. So can you tell me how to create custom storage plugin in this case? Regards, *Anup Tiwari* On Thu, Jan 19, 2017 at 1:55 PM, Nitin Pawar wrote: > you want to use the ORC files created by hive directly in drill or you want > to use them through hive? > > On Thu, Jan 19, 2017 at 1:40 PM, Anup Tiwari > wrote: > > > +Dev > > > > Can someone help me in this? > > > > Regards, > > *Anup Tiwari* > > > > On Sun, Jan 15, 2017 at 2:21 PM, Anup Tiwari > > wrote: > > > > > Hi Team, > > > > > > Can someone tell me how to configure custom storage plugin in Drill for > > > accessing hive ORC tables? > > > > > > Thanks in advance!! > > > > > > Regards, > > > *Anup Tiwari* > > > > > > > > > -- > Nitin Pawar >
