Hi engrdean, I see 2 points here: 1. Log says that load was failed in multi-threaded mode. But then it says that loading was finished, which means that it succeeded in single-threaded mode. 2. Indeed Ignite generates query which is not supported by Hive. And not supported thing is 'mod' operator, which should be '%' in Hive.
If data is loaded fine in single-threaded mode successfully it could be just fine. If you would like to proceed with multi-threaded mode you could try SQLServerDialect as it uses '%' for problematic statement. Unfortunately there is no special Hive dialect in core module. Perhaps you can introduce and even contribute it. Do not hesitate to ask if something is not clear. 2018-07-13 0:31 GMT+03:00 engrdean <engrd...@gmail.com>: > I've been attempting to setup 3rd party persistence with a Hive database > but > I'm running into some issues. I used the Ignite web console to generate > code but when I try to initiate a cache load I receive the error below. > Has > anyone else been successful in using a Hive database with 3rd party > persistence? > > Just to be clear, I am not trying to implement the IgniteHadoopFileSystem > implementation of IGFS. The intent of using 3rd party persistence with > Hive > instead of IGFS is to avoid tightly coupling Ignite with our Hadoop > environment. > > It appears to me that the SQL being generated by Ignite to communicate with > Hive via the driver is probably not something that Hive can parse, but I > would welcome any feedback and I'm happy to provide config files if that > would be helpful. > > [15:02:46,049][INFO][mgmt-#568%test%][CacheJdbcPojoStore] Started load > cache > [cache=EmployeeCache, keyType=com.model.EmployeeKey] > [15:02:46,088][WARNING][mgmt-#568%test%][CacheJdbcPojoStore] Failed to > load > entries from db in multithreaded mode, will try in single thread > [cache=EmployeeCache, keyType=com.model.EmployeeKey] > org.apache.hive.service.cli.HiveSQLException: Error while compiling > statement: FAILED: SemanticException [Error 10011]: Line 1:129 Invalid > function 'mod' > at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:262) > at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils. > java:248) > at > org.apache.hive.jdbc.HiveStatement.runAsyncOnServer( > HiveStatement.java:300) > at > org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241) > at > org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:437) > at > org.apache.hive.jdbc.HivePreparedStatement.executeQuery( > HivePreparedStatement.java:109) > at > com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery( > NewProxyPreparedStatement.java:353) > at > org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.loadCache( > CacheAbstractJdbcStore.java:763) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadCache(GridCacheStoreManagerAdapter. > java:520) > at > org.apache.ignite.internal.processors.cache.distributed. > dht.GridDhtCacheAdapter.localLoadCache(GridDhtCacheAdapter.java:608) > at > org.apache.ignite.internal.processors.cache.GridCacheProxyImpl. > localLoadCache(GridCacheProxyImpl.java:217) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheJob. > localExecute(GridCacheAdapter.java:5520) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$ > LoadCacheJobV2.localExecute(GridCacheAdapter.java:5569) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$ > TopologyVersionAwareJob.execute(GridCacheAdapter.java:6184) > at > org.apache.ignite.compute.ComputeJobAdapter.call( > ComputeJobAdapter.java:132) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2. > execute(GridClosureProcessor.java:1855) > at > org.apache.ignite.internal.processors.job.GridJobWorker$ > 2.call(GridJobWorker.java:566) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6623) > at > org.apache.ignite.internal.processors.job.GridJobWorker. > execute0(GridJobWorker.java:560) > at > org.apache.ignite.internal.processors.job.GridJobWorker. > body(GridJobWorker.java:489) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > org.apache.ignite.internal.processors.job.GridJobProcessor. > processJobExecuteRequest(GridJobProcessor.java:1123) > at > org.apache.ignite.internal.processors.job.GridJobProcessor$ > JobExecutionListener.onMessage(GridJobProcessor.java:1921) > at > org.apache.ignite.internal.managers.communication. > GridIoManager.invokeListener(GridIoManager.java:1555) > at > org.apache.ignite.internal.managers.communication.GridIoManager. > processRegularMessage0(GridIoManager.java:1183) > at > org.apache.ignite.internal.managers.communication. > GridIoManager.access$4200(GridIoManager.java:126) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run( > GridIoManager.java:1090) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10011]: Line 1:129 > Invalid function 'mod' > at > org.apache.hive.service.cli.operation.Operation. > toSQLException(Operation.java:323) > at > org.apache.hive.service.cli.operation.SQLOperation. > prepare(SQLOperation.java:148) > at > org.apache.hive.service.cli.operation.SQLOperation. > runInternal(SQLOperation.java:226) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:264) > at > org.apache.hive.service.cli.session.HiveSessionImpl. > executeStatementInternal(HiveSessionImpl.java:470) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync( > HiveSessionImpl.java:457) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hive.service.cli.session.HiveSessionProxy. > invoke(HiveSessionProxy.java:78) > at > org.apache.hive.service.cli.session.HiveSessionProxy. > access$000(HiveSessionProxy.java:36) > at > org.apache.hive.service.cli.session.HiveSessionProxy$1. > run(HiveSessionProxy.java:63) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1866) > at > org.apache.hive.service.cli.session.HiveSessionProxy. > invoke(HiveSessionProxy.java:59) > at com.sun.proxy.$Proxy26.executeStatementAsync(Unknown Source) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync( > CLIService.java:313) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement( > ThriftCLIService.java:509) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ > ExecuteStatement.getResult(TCLIService.java:1302) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ > ExecuteStatement.getResult(TCLIService.java:1317) > at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process( > TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process( > TSetIpAddressProcessor.java:56) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( > TThreadPoolServer.java:286) > ... 3 more > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.parse.SemanticException:Line 1:129 Invalid > function 'mod' > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor. > getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:936) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$ > DefaultExprProcessor.process(TypeCheckProcFactory.java:1276) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch( > DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn( > DefaultGraphWalker.java:95) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch( > DefaultGraphWalker.java:79) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk( > DefaultGraphWalker.java:133) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking( > DefaultGraphWalker.java:110) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode( > TypeCheckProcFactory.java:214) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode( > TypeCheckProcFactory.java:158) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc( > SemanticAnalyzer.java:10866) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc( > SemanticAnalyzer.java:10822) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc( > SemanticAnalyzer.java:10790) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan( > SemanticAnalyzer.java:2928) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan( > SemanticAnalyzer.java:2909) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer. > genBodyPlan(SemanticAnalyzer.java:8968) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer. > genPlan(SemanticAnalyzer.java:9900) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer. > genPlan(SemanticAnalyzer.java:9779) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer. > genOPTree(SemanticAnalyzer.java:10431) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner. > genOPTree(CalcitePlanner.java:336) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal( > SemanticAnalyzer.java:10442) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal( > CalcitePlanner.java:218) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer. > analyze(BaseSemanticAnalyzer.java:237) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:465) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321) > at > org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1221) > at > org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1215) > at > org.apache.hive.service.cli.operation.SQLOperation. > prepare(SQLOperation.java:146) > ... 27 more > [15:02:46,089][INFO][C3P0PooledConnectionPoolManager[identityToken-> > 1hgf3ss9w88jmqz33q0sc|437a7f8e]-HelperThread-#2][Utils] > Supplied authorities: 127.0.0.1:10000 > [15:02:46,090][INFO][C3P0PooledConnectionPoolManager[identityToken-> > 1hgf3ss9w88jmqz33q0sc|437a7f8e]-HelperThread-#2][Utils] > Resolved authority: 127.0.0.1:10000 > [15:02:46,161][INFO][mgmt-#568%copp%][CacheJdbcPojoStore] Finished load > cache: EmployeeCache > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > -- Best regards, Ivan Pavlukhin
