I use hcatalog as metadata Storage .so I can use hive to create table and
manipulate data .I can also use pig to manipulate the table which Hive create.
now, I can use
"a = load 'tableHIVE' using org.apache.hcatalog.pig.HCatLoader();"
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.input.dir.recursive
is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.max.split.size is
deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.min.split.size is
deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.min.split.size.per.rack is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize.per.rack
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.min.split.size.per.node is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize.per.node
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
2014-12-23 15:31:45,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
2014-12-23 15:31:46,318 [main] INFO hive.metastore - Trying to connect to
metastore with URI thrift://10.210.90.*:9083
2014-12-23 15:31:46,407 [main] INFO hive.metastore - Connected to metastore.
2014-12-23 15:31:46,702 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
but when I store the data . I always get error：
"store a into 'tablePIG' using org.apache.hcatalog.pig.HCatStorer(); "
2014-12-23 15:36:54,051 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2014-12-23 15:36:54,164 [main] INFO hive.metastore - Trying to connect to
metastore with URI thrift://10.210.90.*:9083
2014-12-23 15:36:54,166 [main] INFO hive.metastore - Connected to metastore.
2014-12-23 15:36:54,216 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig features used in the script: UNKNOWN
2014-12-23 15:36:54,218 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite,
GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer,
LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer,
PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter,
StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-12-23 15:36:54,219 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2014-12-23 15:36:54,225 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File
concatenation threshold: 100 optimistic? false
2014-12-23 15:36:54,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2014-12-23 15:36:54,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2014-12-23 15:36:54,255 [main] INFO org.apache.hadoop.yarn.client.RMProxy -
Connecting to ResourceManager at pdmdev00/10.210.90.101:8050
2014-12-23 15:36:54,266 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig script settings are added to the job
2014-12-23 15:36:54,272 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-12-23 15:36:54,373 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2014-12-23 15:36:54,380 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job8750450960344518471.jar
2014-12-23 15:36:59,084 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job8750450960344518471.jar created
2014-12-23 15:36:59,096 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-12-23 15:36:59,145 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2014-12-23 15:36:59,150 [JobControl] INFO
org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at
pdmdev00/10.210.90.101:8050
2014-12-23 15:37:00,123 [JobControl] INFO
org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1
2014-12-23 15:37:00,123 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2014-12-23 15:37:00,376 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2014-12-23 15:37:00,527 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job:
job_1418807862295_0055
2014-12-23 15:37:00,772 [JobControl] INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application
application_1418807862295_0055
2014-12-23 15:37:00,782 [JobControl] INFO org.apache.hadoop.mapreduce.Job -
The url to track the job:
http://pdmdev00:8088/proxy/application_1418807862295_0055/
2014-12-23 15:37:00,783 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_1418807862295_0055
2014-12-23 15:37:00,783 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases a
2014-12-23 15:37:00,783 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: a[1,4] C: R:
2014-12-23 15:37:00,827 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2014-12-23 15:37:15,910 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop
immediately on failure.
2014-12-23 15:37:15,910 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_1418807862295_0055 has failed! Stop running all dependent jobs
2014-12-23 15:37:15,910 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2014-12-23 15:37:15,971 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil
- 1 map reduce job(s) failed!
2014-12-23 15:37:15,971 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersionPigVersionUserIdStartedAtFinishedAtFeatures
2.3.0-bc1.0.00.12.0-cdh5.0.0root2014-12-23 15:36:542014-12-23 15:37:15UNKNOWN
Failed!
Failed Jobs:
JobIdAliasFeatureMessageOutputs
job_1418807862295_0055aMAP_ONLYMessage: Job
failed!hdfs://pdmdev00:8020/tmp/temp418476715/tmp-1157967161,
Input(s):
Failed to read data from "table1418909729552"
Output(s):
Failed to produce result in
"hdfs://pdmdev00:8020/tmp/temp418476715/tmp-1157967161"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1418807862295_0055
2014-12-23 15:37:15,972 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2014-12-23 15:37:15,979 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
1066: Unable to open iterator for alias a
Details at logfile: /root/pig_1419302861874.log
