Hi Fatih, the direct mode is delegating the data transfer task to database native utilities. In case of MySQL it would be mysqldump and/or mysqlimport, for the Postgresql psql is needed. You need to ensure that those utilities are installed on all nodes in the cluster in order to use the direct mode functionality.
I would recommend you taking a look into the Apache Sqoop Cookbook [1] where this any many other details of Sqoop working are described. Jarcec Links: 1: http://shop.oreilly.com/product/0636920029519.do On Tue, Jul 16, 2013 at 03:06:05PM +0400, Fatih Haltas wrote: > Hi everyone, > > I am trying to import data from postgresql to hdfs. But I am having some > problems, Here is the problem details: > > Sqoop Version: 1.4.3 > Hadoop Version:1.0.4 > > > *1) When I use this command:* > * > * > *./sqoop import-all-tables --connect jdbc:postgresql:// > 192.168.194.158:5432/IMS --username pgsql -P* > > > Here is the exact output > ====================================================== > Warning: /usr/lib/hbase does not exist! HBase imports will fail. > Please set $HBASE_HOME to the root of your HBase installation. > Warning: $HADOOP_HOME is deprecated. > > Enter password: > 13/07/16 13:52:28 INFO manager.SqlManager: Using default fetchSize of 1000 > 13/07/16 13:52:29 INFO tool.CodeGenTool: Beginning code generation > 13/07/16 13:52:29 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM "publicinici" AS t LIMIT 1 > 13/07/16 13:52:29 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is > /home/hadoop/project/hadoop-1.0.4 > Note: > /tmp/sqoop-hadoop/compile/0f484159ce27d8ed3c2d95ca13974f1a/publicinici.java > uses or overrides a deprecated API. > Note: Recompile with -Xlint:deprecation for details. > 13/07/16 13:52:30 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-hadoop/compile/0f484159ce27d8ed3c2d95ca13974f1a/publicinici.jar > 13/07/16 13:52:30 WARN manager.PostgresqlManager: It looks like you are > importing from postgresql. > 13/07/16 13:52:30 WARN manager.PostgresqlManager: This transfer can be > faster! Use the --direct > 13/07/16 13:52:30 WARN manager.PostgresqlManager: option to exercise a > postgresql-specific fast path. > 13/07/16 13:52:30 INFO mapreduce.ImportJobBase: Beginning import of > publicinici > 13/07/16 13:52:31 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: > SELECT MIN("budapablik"), MAX("budapablik") FROM "publicinici" > 13/07/16 13:52:31 INFO mapred.JobClient: Running job: job_201302261146_0525 > 13/07/16 13:52:32 INFO mapred.JobClient: map 0% reduce 0% > ^X[hadoop@ADUAE042-LAP-V bin]./sqoop version > Warning: /usr/lib/hbase does not exist! HBase imports will fail. > Please set $HBASE_HOME to the root of your HBase installation. > Warning: $HADOOP_HOME is deprecated. > =================================================== > > *I have only one table which is "publicinici",that holds 2 entity in one row > * and it read but at mapreduce phase it is getting stucked. > > > *2) When I use this command in which --direct option is added, eveything is > going well, but getting one error, which is * > > ERROR tool.ImportAllTablesTool: Encountered IOException running import job: > java.io.IOException: Cannot run program "psql": java.io.IOException: > error=2, No such file or directory > > *Also, in the hdfs publicinici file has been created as empty. * > > Here is the exact output. > ======================== > 13/07/16 13:58:42 INFO manager.SqlManager: Using default fetchSize of 1000 > 13/07/16 13:58:42 INFO tool.CodeGenTool: Beginning code generation > 13/07/16 13:58:43 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM "publicinici" AS t LIMIT 1 > 13/07/16 13:58:43 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is > /home/hadoop/project/hadoop-1.0.4 > Note: > /tmp/sqoop-hadoop/compile/56c2a5b04e83ace0112fe038eb4d1599/publicinici.java > uses or overrides a deprecated API. > Note: Recompile with -Xlint:deprecation for details. > 13/07/16 13:58:44 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-hadoop/compile/56c2a5b04e83ace0112fe038eb4d1599/publicinici.jar > 13/07/16 13:58:44 INFO manager.DirectPostgresqlManager: Beginning psql fast > path import > 13/07/16 13:58:44 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM "publicinici" AS t LIMIT 1 > 13/07/16 13:58:44 INFO manager.DirectPostgresqlManager: Copy command is > COPY (SELECT "budapablik" FROM "publicinici" WHERE 1=1) TO STDOUT WITH > DELIMITER E'\54' CSV ; > 13/07/16 13:58:44 INFO manager.DirectPostgresqlManager: Performing import > of table publicinici from database IMS > 13/07/16 13:58:45 INFO manager.DirectPostgresqlManager: Transfer loop > complete. > 13/07/16 13:58:45 INFO manager.DirectPostgresqlManager: Transferred 0 bytes > in 17,361,268.2044 seconds (0 bytes/sec) > 13/07/16 13:58:45 ERROR tool.ImportAllTablesTool: Encountered IOException > running import job: java.io.IOException: Cannot run program "psql": > java.io.IOException: error=2, No such file or directory > > > What should I do to be able to fix the problem. > Any help is gonna be appreciated. Thank you very much.
signature.asc
Description: Digital signature
