A follow up
If I insert into a local file and then load the file into the table
using a load command it works:
hive> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/hdfs_out' SELECT
TRANSFORM(actor_id) USING '/Users/Josh/percentiles.rb' AS (actor_id,
percentile, count) FROM (SELECT actor_id FROM activities CLUSTER BY
actor_id) actor;
hive> LOAD DATA LOCAL INPATH '/tmp/hdfs_out/
attempt_200901112100_0030_r_000000_0' OVERWRITE INTO TABLE percentiles
PARTITION ( account='cUU5T7y6DmdzMJFcFt3JDe', application='Test
Application', dataset='Purchases', hour=342007, span=1 );
Copying data from file:/tmp/hdfs_out/
attempt_200901112100_0030_r_000000_0
Loading data to table percentiles partition
{account=cUU5T7y6DmdzMJFcFt3JDe, application=Test Application,
dataset=Purchases, hour=342007, span=1}
OK
hive> SELECT * FROM percentiles;
OK
00d5c3f0-1b29-4bd3-bc8c-e3f0a7ba5949 100 1 cUU5T7y6DmdzMJFcFt3JDe Test
Application Purchases 342007 1
03e58605-7de7-48fb-9852-781700823a71 100 1 cUU5T7y6DmdzMJFcFt3JDe Test
Application Purchases 342007 1
06986e3e-7c73-4466-b0dc-d92038e3f665 100 1 cUU5T7y6DmdzMJFcFt3JDe Test
Application Purchases 342007 1
06af307c-0da6-4795-860a-6b26425bdbc8 100 1 cUU5T7y6DmdzMJFcFt3JDe Test
Application Purchases 342007 1
0cca5b18-efe3-4903-9387-a8c7eb8432e4 100 1 cUU5T7y6DmdzMJFcFt3JDe Test
Application Purchases 342007 1
0e20b565-59aa-4d57-8f68-ddcaaa75673b 100 1 cUU5T7y6DmdzMJFcFt3JDe Test
Application Purchases 342007 1
...
So it works in two parts but not in one part, I'm not sure why.
Josh F.
On Jan 12, 2009, at 9:57 PM, Zheng Shao wrote:
Does the map-reduce job produced any output? Can you check the
mapred job page?
If so then it must be that the "moveTask" after map-reduce job is
failed.
Zheng
On Mon, Jan 12, 2009 at 9:48 PM, Josh Ferguson <j...@besquared.net>
wrote:
https://gist.github.com/3cb4be29625442c90140
Josh
On Jan 12, 2009, at 9:39 PM, Zheng Shao wrote:
Should be tab separated.
if you run it without insert, is there any data on the screen?
Zheng
On Mon, Jan 12, 2009 at 9:32 PM, Josh Ferguson <j...@besquared.net>
wrote:
The only thing I could figure is that my output is incorrect.. Is
the output from the transform script supposed to be tab separated
or separated by the delimiters of the table you're trying to insert
into? It doesn't seem to make any difference (my table is still
empty no matter which one I try) but I'd better make sure just
incase.
Josh
On Jan 12, 2009, at 9:11 PM, Zheng Shao wrote:
Here are some examples:
[zs...@xxx /hive.root] find ./ql/src/test/queries/clientpositive -
name '*.q' | xargs grep TRANSFORM
./ql/src/test/queries/clientpositive/input14_limit.q: SELECT
TRANSFORM(src.key, src.value)
./ql/src/test/queries/clientpositive/input14_limit.q: SELECT
TRANSFORM(src.key, src.value)
./ql/src/test/queries/clientpositive/input14.q: SELECT
TRANSFORM(src.key, src.value)
./ql/src/test/queries/clientpositive/input14.q: SELECT
TRANSFORM(src.key, src.value)
./ql/src/test/queries/clientpositive/input18.q: SELECT
TRANSFORM(src.key, src.value, 1+2, 3+4)
./ql/src/test/queries/clientpositive/input18.q: SELECT
TRANSFORM(src.key, src.value, 1+2, 3+4)
./ql/src/test/queries/clientpositive/scriptfile1.q: SELECT
TRANSFORM(src.key, src.value)
./ql/src/test/queries/clientpositive/input5.q: SELECT
TRANSFORM(src_thrift.lint, src_thrift.lintstring)
./ql/src/test/queries/clientpositive/input5.q: SELECT
TRANSFORM(src_thrift.lint, src_thrift.lintstring)
./ql/src/test/queries/clientpositive/input17.q: SELECT
TRANSFORM(src_thrift.aint + src_thrift.lint[0],
src_thrift.lintstring[0])
./ql/src/test/queries/clientpositive/input17.q: SELECT
TRANSFORM(src_thrift.aint + src_thrift.lint[0],
src_thrift.lintstring[0])
On Mon, Jan 12, 2009 at 8:58 PM, Josh Ferguson
<j...@besquared.net> wrote:
Anyone have any word on why this might not work? Can someone give
me an example of a query they use to INSERT OVERWRITE a table from
a map and/or reduce job that I could use as a reference?
Josh F.
On Jan 11, 2009, at 9:48 PM, Josh Ferguson wrote:
I have a query that returns the proper results:
SELECT TRANSFORM(actor_id) USING '/my/script.rb' AS (actor_id,
percentile, count) FROM (SELECT actor_id FROM activities CLUSTER
BY actor_id) actors;
But when I do
INSERT OVERWRITE TABLE percentiles
SELECT TRANSFORM(actor_id) USING '/my/script.rb' AS (actor_id,
percentile, count) FROM (SELECT actor_id FROM activities CLUSTER
BY actor_id) actors;
It says it loads data into the percentiles table but when I ask
for data from that table I get:
hive> SELECT actor_id, percentile, count FROM percentiles;
FAILED: Error in semantic analysis:
org.apache.hadoop.hive.ql.metadata.HiveException: Path /user/hive/
warehouse/percentiles not a valid path
$ hadoop fs -ls /user/hive/warehouse/percentiles/
Found 1 items
-rw-r--r-- 1 Josh supergroup 0 2009-01-11 21:45 /user/
hive/warehouse/percentiles/attempt_200901112100_0010_r_000000_0
It's nothing but an empty file.
Am I doing something wrong?
Josh Ferguson
--
Yours,
Zheng
--
Yours,
Zheng
--
Yours,
Zheng
