Thanks all for the advice, but nothing works for me. If it works for you, do you use an official Spark build or your own custom build with possible bug fixes that make it work? I found this bug in Spark that could cause my issue: https://issues.apache.org/jira/browse/SPARK-9686. I'll let you know if the proposed there fix works.
Where should I look next? Do you think it is a bug in Calcite or Spark? If you think it's not a bug, can you describe your set up that works? For example, do you use default Hive metastore that theoretically should work as described here: https://community.hortonworks.com/questions/23797/how-the-hive-metastore-works.html Or do you use custom metastore, as described here: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration Thanks, Val On Wed, Jun 19, 2019 at 7:20 PM Zhu Feng <[email protected]> wrote: > In our environment (Hive-2.2x), the following configuration works well. > (jdbcCatalog=null, jdbcSchema=db_name) > Therefore, maybe you need to add a line in your Shema's json > .......................... > jdbcSchema: 'thrift' > ............................. > > Best, > DonnyZone > > Valeriy Trofimov <[email protected]> 于2019年6月19日周三 下午11:41写道： > > > Hi Zhu, > > > > Setting it to null shows "database null not found". My schema is missing > > jdbcCatalog entry. What value should I put there? I could not find > anything > > in hive doc: https://cwiki.apache.org/confluence/display/Hive/HiveClient > > > > After playing with it more I've realized that it is able to connect to > > Hive, but not query it (sorry for bad title). > > > > When I run the > > select * from "thrift"."test" > > query it shows the below error. I put quotes because Calcite is case > > sensitive. The error does not mean that table "test" not found in Hive > DB, > > but that the schema doesn't have the "test" element inside the "thrift" > > schema. I think it fails to recognize that what I mean is that I want to > > use "thrift" schema to connect to the database "default" and its table > > "test". For some reason it thinks that "test" is part of the schema, not > > table in the DB. > > > > 07:59:31.399 [main] DEBUG > > com.huawei.cloudsop.us.queryengine.Controller.QueryController - query: > > select * from "thrift"."test" > > 07:59:32.479 [main] INFO org.apache.hive.jdbc.Utils - Supplied > authorities: > > 127.0.0.1:10000 > > 07:59:32.480 [main] INFO org.apache.hive.jdbc.Utils - Resolved authority: > > 127.0.0.1:10000 > > 07:59:32.523 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > opening transport > org.apache.thrift.transport.TSaslClientTransport@19542407 > > 07:59:32.524 [main] DEBUG > org.apache.thrift.transport.TSaslClientTransport > > - Sending mechanism name PLAIN and initial response of length 20 > > 07:59:32.526 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Writing message with status START and payload length 5 > > 07:59:32.526 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Writing message with status COMPLETE and payload length 20 > > 07:59:32.526 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Start message handled > > 07:59:32.526 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Main negotiation loop complete > > 07:59:32.526 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: SASL Client receiving last message > > 07:59:32.526 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Received message with status COMPLETE and payload length 0 > > 07:59:32.538 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 71 > > 07:59:32.575 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 109 > > 07:59:32.710 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 85 > > 07:59:32.714 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 55 > > 07:59:32.723 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 83 > > 07:59:32.724 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 40 > > 07:59:32.733 [main] INFO org.apache.hive.jdbc.Utils - Supplied > authorities: > > 127.0.0.1:10000 > > 07:59:32.733 [main] INFO org.apache.hive.jdbc.Utils - Resolved authority: > > 127.0.0.1:10000 > > 07:59:32.734 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > opening transport > org.apache.thrift.transport.TSaslClientTransport@49798e84 > > 07:59:32.734 [main] DEBUG > org.apache.thrift.transport.TSaslClientTransport > > - Sending mechanism name PLAIN and initial response of length 20 > > 07:59:32.734 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Writing message with status START and payload length 5 > > 07:59:32.734 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Writing message with status COMPLETE and payload length 20 > > 07:59:32.735 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Start message handled > > 07:59:32.735 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Main negotiation loop complete > > 07:59:32.735 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: SASL Client receiving last message > > 07:59:32.736 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: Received message with status COMPLETE and payload length 0 > > 07:59:32.736 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 71 > > 07:59:32.776 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 109 > > 07:59:32.776 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 85 > > 07:59:32.777 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 55 > > 07:59:32.779 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 85 > > 07:59:32.780 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 55 > > 07:59:32.780 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 85 > > 07:59:32.781 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 51 > > 07:59:32.781 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 85 > > 07:59:32.781 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 55 > > 07:59:33.119 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 85 > > 07:59:33.123 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 55 > > 07:59:33.127 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 88 > > 07:59:33.137 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 102 > > 07:59:33.153 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 102 > > 07:59:33.155 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 464 > > 07:59:33.178 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 112 > > 07:59:33.181 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 180 > > 07:59:33.204 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > writing data length: 96 > > 07:59:33.207 [main] DEBUG org.apache.thrift.transport.TSaslTransport - > > CLIENT: reading data length: 42 > > java.sql.SQLException: Error while executing SQL "select * from > > "thrift"."test"": From line 1, column 15 to line 1, column 29: Object > > 'test' not found within 'thrift' > > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > > at > > > > > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163) > > at > > > > > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) > > at > > com.huawei.cloudsop.us > > > .queryengine.Manager.CalciteQueryManager.query(CalciteQueryManager.java:41) > > at > > com.huawei.cloudsop.us > > .queryengine.Controller.QueryController.query(QueryController.java:27) > > at > > com.huawei.cloudsop.us > > .queryengine.SchemaTest.testThriftScheme(SchemaTest.java:41) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > at > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > > at > > > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > > at > > > > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > > at > > > > > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > > at > > > > > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > > at > > > > > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > > Caused by: org.apache.calcite.runtime.CalciteContextException: From line > 1, > > column 15 to line 1, column 29: Object 'test' not found within 'thrift' > > at > > > > > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) > > at > > > > > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > > > > java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > > java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) > > at > > > org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) > > at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787) > > at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825) > > at > > > > > org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:166) > > at > > > > > org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:177) > > at > > > > > org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3087) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3069) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3339) > > at > > > > > org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) > > at > > > > > org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954) > > at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929) > > at > > > > > org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633) > > at > > > > > org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:558) > > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:265) > > at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231) > > at > > > > > org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:767) > > at > > > > > org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:631) > > at > > > > > org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:601) > > at > > > > > org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:229) > > at > > > > > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550) > > at > > > > > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > > at > > > > > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > > ... 26 more > > Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Object > > 'test' not found within 'thrift' > > at > > > > > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) > > at > > > > > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > > > > java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > > java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) > > at > > > org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) > > at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) > > ... 54 more > > > > Thanks, > > Val > > > > > > On Tue, Jun 18, 2019 at 7:39 PM Zhu Feng <[email protected]> wrote: > > > > > Hi Valeriy, > > > I remember that "jdbcCatalog" and "jdbcSchema" should be carefully > > > configured for Hive Jdbc. > > > Maybe you can try "null" or db name. > > > > > > best, > > > DonnyZone > > > > > > Valeriy Trofimov <[email protected]> 于2019年6月19日周三 上午8:11写道： > > > > > > > Hi All, > > > > > > > > I'm trying to connect to Hive DB via Thrift JDBC Server. > > > > > > > > I get error: "Object 'test' not found within 'thrift'". > > > > > > > > The error occurs by the IdentifierNamespace.resolveImpl() method. The > > > > method checks that schema looks correct based on my query. > > > > > > > > Schema: > > > > > > > > { > > > > version: '1.0', > > > > schemas: [ > > > > { > > > > name: 'thrift', > > > > type: 'custom', > > > > factory: 'org.apache.calcite.adapter.jdbc.JdbcSchema$Factory', > > > > operand: { > > > > jdbcDriver: 'org.apache.hive.jdbc.HiveDriver', > > > > jdbcUrl: 'jdbc:hive2://127.0.0.1:10000/default', // please > use > > > > VirtualBox port forwarding to the VM with Thrift server running > > > > jdbcUser: '', > > > > jdbcPassword: '' > > > > } > > > > } > > > > ] > > > > } > > > > > > > > Test query: > > > > > > > > String query = "select * from \"thrift\".\"test\""; > > > > > > > > I've double checked that you can connect to Thrift Server and that > > there > > > is > > > > a database called "default" and table called "test". > > > > > > > > It looks like the error is caused by Calcite trying to find entries > for > > > > "tables" in the schema. But I am confused why it would look for it. > > JDBC > > > > schema doesnt require these entries and adding them actually shows > > error > > > > about "tables" not needing to be there. > > > > > > > > Repro steps for this issue are too long, but I can provide if someone > > is > > > > interested in digging deeper into this. > > > > > > > > Thanks, > > > > Val > > > > > > > > > >
