Hi Gerrit,
Yeah Hive table isn't stored as RCFILE but TEXTFILE
so our table creation ddl looks like below
CREATE EXTERNAL TABLE page_view(viewTime INT, userid BIGINT,
page_url STRING, referrer_url STRING,
ip STRING COMMENT 'IP Address of the User',
country STRING COMMENT 'country of origination')
COMMENT 'This is the staging page view table'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
Does AllLoader understand notion of partition keys? as HiveColumnarLoader?
J
On 1 Dec 2010, at 13:48, Gerrit Jansen van Vuuren wrote:
> Hi,
>
> The HiveColumnarLoader can only read files written by hive or the hive
> API(s), and has its own InputFormat returning the HiveRCRecordReader.
>
> Are you trying to read a plain text format?
> Under the hood the HiveRCRecordReader uses the hive specific rc reader to
> read the input file and throws an error either if the file is not hive rc or
> is a corrupt hiverc.
>
>
> If what you want is a Loader that loads all types of files, have a look at
> the AllLoader (latest piggybank trunk). It uses configuration that you set
> in the pig.properties to decide on the fly what loader to use for what files
> (does extension, content and path matching), it also has the hive style path
> partitioning for dates etc. Using this loader you can point it at a directoy
> with lzo, gz, bz2 hiverc etc files in it and if you setup the loaders
> correctly it will load each file with its preconfigured loader.
> The javadoc in the class explains how to configure it.
>
> Cheers,
> Gerrit
>
> -----Original Message-----
> From: Jae Lee [mailto:jae....@forward.co.uk]
> Sent: Wednesday, December 01, 2010 12:33 PM
> To: dev@pig.apache.org
> Subject: has anyone tried using HiveColumnarLoader over TextFile fileformat?
>
> Hi everyone.
>
> I've tried using HiveColumnarLoader and getting java.io.IOException:
> hdfs://file_path not a RCFile
>
> I've noticed HiveColumnarLoader is expecting HiveRCRecordReader from
> prepareToRead method..
>
> Could you guys give any guidance how possible it is to modify
> HiveRCRecordReader to support any RecordReader?
>
> J
>
>
