I think it is the same problem as Jeremy Chow's (the name configured in hadoop-site.xml and hive-default.xml seem to be different).
________________________________ From: Ashish Thusoo <athu...@facebook.com> Reply-To: <hive-user@hadoop.apache.org> Date: Tue, 13 Jan 2009 11:30:18 -0800 To: <hive-user@hadoop.apache.org> Subject: RE: INSERT OVERWRITE not working with map/reduce transform Can you also post the output of explain <query> Thanks, Ashish ________________________________ From: Josh Ferguson [mailto:j...@besquared.net] Sent: Monday, January 12, 2009 10:24 PM To: hive-user@hadoop.apache.org Subject: Re: INSERT OVERWRITE not working with map/reduce transform A follow up If I insert into a local file and then load the file into the table using a load command it works: hive> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/hdfs_out' SELECT TRANSFORM(actor_id) USING '/Users/Josh/percentiles.rb' AS (actor_id, percentile, count) FROM (SELECT actor_id FROM activities CLUSTER BY actor_id) actor; hive> LOAD DATA LOCAL INPATH '/tmp/hdfs_out/attempt_200901112100_0030_r_000000_0' OVERWRITE INTO TABLE percentiles PARTITION ( account='cUU5T7y6DmdzMJFcFt3JDe', application='Test Application', dataset='Purchases', hour=342007, span=1 ); Copying data from file:/tmp/hdfs_out/attempt_200901112100_0030_r_000000_0 Loading data to table percentiles partition {account=cUU5T7y6DmdzMJFcFt3JDe, application=Test Application, dataset=Purchases, hour=342007, span=1} OK hive> SELECT * FROM percentiles; OK 00d5c3f0-1b29-4bd3-bc8c-e3f0a7ba5949 100 1 cUU5T7y6DmdzMJFcFt3JDe Test Application Purchases 342007 1 03e58605-7de7-48fb-9852-781700823a71 100 1 cUU5T7y6DmdzMJFcFt3JDe Test Application Purchases 342007 1 06986e3e-7c73-4466-b0dc-d92038e3f665 100 1 cUU5T7y6DmdzMJFcFt3JDe Test Application Purchases 342007 1 06af307c-0da6-4795-860a-6b26425bdbc8 100 1 cUU5T7y6DmdzMJFcFt3JDe Test Application Purchases 342007 1 0cca5b18-efe3-4903-9387-a8c7eb8432e4 100 1 cUU5T7y6DmdzMJFcFt3JDe Test Application Purchases 342007 1 0e20b565-59aa-4d57-8f68-ddcaaa75673b 100 1 cUU5T7y6DmdzMJFcFt3JDe Test Application Purchases 342007 1 ... So it works in two parts but not in one part, I'm not sure why. Josh F. On Jan 12, 2009, at 9:57 PM, Zheng Shao wrote: Does the map-reduce job produced any output? Can you check the mapred job page? If so then it must be that the "moveTask" after map-reduce job is failed. Zheng On Mon, Jan 12, 2009 at 9:48 PM, Josh Ferguson <j...@besquared.net> wrote: https://gist.github.com/3cb4be29625442c90140 Josh On Jan 12, 2009, at 9:39 PM, Zheng Shao wrote: Should be tab separated. if you run it without insert, is there any data on the screen? Zheng On Mon, Jan 12, 2009 at 9:32 PM, Josh Ferguson <j...@besquared.net> wrote: The only thing I could figure is that my output is incorrect.. Is the output from the transform script supposed to be tab separated or separated by the delimiters of the table you're trying to insert into? It doesn't seem to make any difference (my table is still empty no matter which one I try) but I'd better make sure just incase. Josh On Jan 12, 2009, at 9:11 PM, Zheng Shao wrote: Here are some examples: [zs...@xxx /hive.root] find ./ql/src/test/queries/clientpositive -name '*.q' | xargs grep TRANSFORM ./ql/src/test/queries/clientpositive/input14_limit.q: SELECT TRANSFORM(src.key, src.value) ./ql/src/test/queries/clientpositive/input14_limit.q: SELECT TRANSFORM(src.key, src.value) ./ql/src/test/queries/clientpositive/input14.q: SELECT TRANSFORM(src.key, src.value) ./ql/src/test/queries/clientpositive/input14.q: SELECT TRANSFORM(src.key, src.value) ./ql/src/test/queries/clientpositive/input18.q: SELECT TRANSFORM(src.key, src.value, 1+2, 3+4) ./ql/src/test/queries/clientpositive/input18.q: SELECT TRANSFORM(src.key, src.value, 1+2, 3+4) ./ql/src/test/queries/clientpositive/scriptfile1.q: SELECT TRANSFORM(src.key, src.value) ./ql/src/test/queries/clientpositive/input5.q: SELECT TRANSFORM(src_thrift.lint, src_thrift.lintstring) ./ql/src/test/queries/clientpositive/input5.q: SELECT TRANSFORM(src_thrift.lint, src_thrift.lintstring) ./ql/src/test/queries/clientpositive/input17.q: SELECT TRANSFORM(src_thrift.aint + src_thrift.lint[0], src_thrift.lintstring[0]) ./ql/src/test/queries/clientpositive/input17.q: SELECT TRANSFORM(src_thrift.aint + src_thrift.lint[0], src_thrift.lintstring[0]) On Mon, Jan 12, 2009 at 8:58 PM, Josh Ferguson <j...@besquared.net> wrote: Anyone have any word on why this might not work? Can someone give me an example of a query they use to INSERT OVERWRITE a table from a map and/or reduce job that I could use as a reference? Josh F. On Jan 11, 2009, at 9:48 PM, Josh Ferguson wrote: I have a query that returns the proper results: SELECT TRANSFORM(actor_id) USING '/my/script.rb' AS (actor_id, percentile, count) FROM (SELECT actor_id FROM activities CLUSTER BY actor_id) actors; But when I do INSERT OVERWRITE TABLE percentiles SELECT TRANSFORM(actor_id) USING '/my/script.rb' AS (actor_id, percentile, count) FROM (SELECT actor_id FROM activities CLUSTER BY actor_id) actors; It says it loads data into the percentiles table but when I ask for data from that table I get: hive> SELECT actor_id, percentile, count FROM percentiles; FAILED: Error in semantic analysis: org.apache.hadoop.hive.ql.metadata.HiveException: Path /user/hive/warehouse/percentiles not a valid path $ hadoop fs -ls /user/hive/warehouse/percentiles/ Found 1 items -rw-r--r-- 1 Josh supergroup 0 2009-01-11 21:45 /user/hive/warehouse/percentiles/attempt_200901112100_0010_r_000000_0 It's nothing but an empty file. Am I doing something wrong? Josh Ferguson -- Yours, Zheng -- Yours, Zheng -- Yours, Zheng
