Hi, this pr is still under review, it is recommended to try after merge to master, we will complete the review and merge in these days.
> 在 2019年10月28日，15:08，Andrey Molotov <[email protected]> 写道： > > Hello, thank you for your answer. > I pulled the commit you provided and compiled jar file (two jar files, > actually: kylin-source-jdbc-3.0.0-SNAPSHOT.jar and > kylin-jdbc-3.0.0-SNAPSHOT.jar). Then for each of these files I did following: > renamed it and put it instead of existing kylin-jdbc-2.6.4.jar file in > kylin/lib directory. > But unfortunately this did help me resolve my problem with the backtick in > SQL query. > Is there any other way to get a proper query line for PostgreSQL or maybe I > did something wrong? Thanks in advance. > >> 16 окт. 2019 г., в 02:51, "[email protected]" <[email protected]> >> написал(а): >> >> Hi, Molotov, because postgresql's syntax and metadata have certain >> specialities, need to do some development work. PR >> https://github.com/apache/kylin/pull/747 >> <https://github.com/apache/kylin/pull/747> id doing this kind of thing, it >> is in review now. >> >>>> 在 2019年10月15日，20:54，Andrey Molotov <[email protected]> 写道： >>> Hello, everyone. >>> I’ve set up Kylin to access a PostgreSQL Database using JDBC as described >>> in http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html . >>> I’ve also set kylin.source.default=16 and >>> kylin.source.hive.enable.quote=false in kylin.properties. >>> But when I try to build a cube a get an error on #1 Step Name: Sqoop To >>> Flat Hive Table. >>> My Kylin Version is 2.6.4. >>> Here is log: >>> java.io.IOException: OS command error exit with return code: 1, >>> error message: Error: Could not find or load main class >>> org.apache.hadoop.hbase.util.GetJavaProperty >>> SLF4J: Class path contains multiple SLF4J bindings. >>> SLF4J: Found binding in >>> [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: Found binding in >>> [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: Found binding in >>> [jar:file:/opt/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>> explanation. >>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>> 2019-10-15 08:40:23,908 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 >>> 2019-10-15 08:40:23,936 WARN tool.BaseSqoopTool: Setting your password on >>> the command-line is insecure. Consider using -P instead. >>> 2019-10-15 08:40:24,004 WARN sqoop.ConnFactory: Parameter --driver is set >>> to an explicit driver however appropriate connection manager is not being >>> set (via --connection-manager). Sqoop is going to fall back to >>> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly >>> which connection manager should be used next time. >>> 2019-10-15 08:40:24,017 INFO manager.SqlManager: Using default fetchSize of >>> 1000 >>> 2019-10-15 08:40:24,017 INFO tool.CodeGenTool: Beginning code generation >>> 2019-10-15 08:40:24,164 INFO manager.SqlManager: Executing SQL statement: >>> SELECT "installations"."city" AS "INSTALLATIONS_CITY", >>> "installations"."device_type" AS "INSTALLATIONS_DEVICE_TYPE", >>> "installations"."install_datetime" >>> FROM "data"."installations" AS "installations" >>> WHERE 1 = 1 AND ("installations"."install_datetime" >= '2019-01-01' AND >>> "installations"."install_datetime" < '2019-01-03') AND (1 = 0) >>> 2019-10-15 08:40:24,176 INFO manager.SqlManager: Executing SQL statement: >>> SELECT "installations"."city" AS "INSTALLATIONS_CITY", >>> "installations"."device_type" AS "INSTALLATIONS_DEVICE_TYPE", >>> "installations"."install_datetime" >>> FROM "data"."installations" AS "installations" >>> WHERE 1 = 1 AND ("installations"."install_datetime" >= '2019-01-01' AND >>> "installations"."install_datetime" < '2019-01-03') AND (1 = 0) >>> 2019-10-15 08:40:24,200 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is >>> /opt/hadoop >>> Note: >>> /tmp/sqoop-hadoop/compile/33bbb7f633bb5f8338ed0a8e1e7ce3cc/QueryResult.java >>> uses or overrides a deprecated API. >>> Note: Recompile with -Xlint:deprecation for details. >>> 2019-10-15 08:40:25,545 INFO orm.CompilationManager: Writing jar file: >>> /tmp/sqoop-hadoop/compile/33bbb7f633bb5f8338ed0a8e1e7ce3cc/QueryResult.jar >>> 2019-10-15 08:40:25,564 INFO mapreduce.ImportJobBase: Beginning query >>> import. >>> 2019-10-15 08:40:25,565 INFO Configuration.deprecation: mapred.job.tracker >>> is deprecated. Instead, use mapreduce.jobtracker.address >>> 2019-10-15 08:40:25,650 INFO Configuration.deprecation: mapred.jar is >>> deprecated. Instead, use mapreduce.job.jar >>> 2019-10-15 08:40:26,327 INFO Configuration.deprecation: mapred.map.tasks is >>> deprecated. Instead, use mapreduce.job.maps >>> 2019-10-15 08:40:26,392 INFO client.RMProxy: Connecting to ResourceManager >>> at /0.0.0.0:8032 >>> 2019-10-15 08:40:26,640 INFO mapreduce.JobResourceUploader: Disabling >>> Erasure Coding for path: >>> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1571114338286_0004 >>> 2019-10-15 08:40:33,776 INFO db.DBInputFormat: Using read commited >>> transaction isolation >>> 2019-10-15 08:40:33,777 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: >>> SELECT MIN("installations"."install_datetime"), >>> MAX("installations"."install_datetime") >>> FROM "data"."installations" AS "installations" WHERE >>> `INSTALLATIONS`.`INSTALL_DATETIME` >= '2019-01-01' AND >>> `INSTALLATIONS`.`INSTALL_DATETIME` < '2019-01-03' >>> 2019-10-15 08:40:33,785 INFO mapreduce.JobSubmitter: Cleaning up the >>> staging area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1571114338286_0004 >>> 2019-10-15 08:40:33,789 ERROR tool.ImportTool: Import failed: >>> java.io.IOException: org.postgresql.util.PSQLException: ERROR: syntax error >>> at or near "." >>> Position: 164 >>> at >>> org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:207) >>> at >>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:310) >>> at >>> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:327) >>> at >>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200) >>> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) >>> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) >>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) >>> at >>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588) >>> at >>> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:200) >>> at >>> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:173) >>> at >>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:270) >>> at >>> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:748) >>> at >>> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:522) >>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) >>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) >>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) >>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) >>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) >>> at org.apache.sqoop.Sqoop.main(Sqoop.java:252) >>> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or >>> near "." >>> Position: 164 >>> at >>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2497) >>> at >>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2233) >>> at >>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310) >>> at >>> org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) >>> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) >>> at >>> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) >>> at >>> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) >>> at >>> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) >>> at >>> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) >>> at >>> org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:178) >>> ... 22 more >>> The command is: >>> /opt/sqoop/bin/sqoop import >>> -Dorg.apache.sqoop.splitter.allow_text_splitter=true >>> -Dmapreduce.job.queuename=default --connect >>> "jdbc:postgresql://172.0.0.1:8080/database" --driver org.postgresql.Driver >>> --username test --password test --query "SELECT \"installations\".\"city\" >>> AS \"INSTALLATIONS_CITY\", \"installations\".\"device_type\" AS >>> \"INSTALLATIONS_DEVICE_TYPE\", \"installations\".\"install_datetime\" >>> FROM \"data\".\"installations\" AS \"installations\" >>> WHERE 1 = 1 AND (\"installations\".\"install_datetime\" >= '2019-01-01' AND >>> \"installations\".\"install_datetime\" < '2019-01-03') AND \$CONDITIONS" >>> --target-dir >>> hdfs://localhost:9000/kylin/kylin_metadata/kylin-530cd3bb-f258-6ff8-9c38-9514a2258848/kylin_intermediate_7_5c51bb9d_b1ad_1135_a08f_12139ba64070 >>> --split-by \"installations\".\"install_datetime\" --boundary-query "SELECT >>> MIN(\"installations\".\"install_datetime\"), >>> MAX(\"installations\".\"install_datetime\") >>> FROM \"data\".\"installations\" AS \"installations\" WHERE >>> \`INSTALLATIONS\`.\`INSTALL_DATETIME\` >= '2019-01-01' AND >>> \`INSTALLATIONS\`.\`INSTALL_DATETIME\` < '2019-01-03'" --null-string '' >>> --fields-terminated-by '|' --num-mappers 4 >>> at >>> org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96) >>> at >>> org.apache.kylin.source.jdbc.sqoop.SqoopCmdStep.sqoopFlatHiveTable(SqoopCmdStep.java:59) >>> at >>> org.apache.kylin.source.jdbc.sqoop.SqoopCmdStep.doWork(SqoopCmdStep.java:70) >>> at >>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179) >>> at >>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) >>> at >>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179) >>> at >>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Please, help me resolve the problem.
