Here's the stack trace (below). What's interesting is I just verified a job
with this filter statement works:
f = FILTER data BY part_dt == '20120313T000000Z';
Then the stack trace below happens when changing to:
f = FILTER data BY part_dt >= '20120313T000000Z';
This suggests its not a thrift version issue, and I've double-checked and
everything appears to be thrift 0.7.0 (looked inside jars for potential
packaging-related conflicts too).
Additionally, here's a similar show partitions statement in the Hive shell:
hive> show partitions foo partition(part_dt='20120313T110000Z');
OK
part_dt=20120313T110000Z
Time taken: 0.257 seconds
hive> show partitions foo partition(part_dt>='20120313T110000Z');
FAILED: Parse Error: line 1:39 mismatched input '>=' expecting ) near
'part_dt' in show statement
hive> show partitions foo partition(part_dt>'20120313T110000Z');
FAILED: Parse Error: line 1:39 mismatched input '>' expecting ) near
'part_dt' in show statement
Would you expect this sort of range query to work?
Pig Stack Trace
---------------
ERROR 2017: Internal error creating job configuration.
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias l
at org.apache.pig.PigServer.openIterator(PigServer.java:857)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:561)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias l
at org.apache.pig.PigServer.storeEx(PigServer.java:956)
at org.apache.pig.PigServer.store(PigServer.java:919)
at org.apache.pig.PigServer.openIterator(PigServer.java:832)
... 12 more
Caused by:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
ERROR 2017: Internal error creating job configuration.
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:736)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:261)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:151)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
at org.apache.pig.PigServer.storeEx(PigServer.java:952)
... 14 more
Caused by: java.io.IOException:
org.apache.thrift.transport.TTransportException
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:42)
at
org.apache.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:90)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385)
... 19 more
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_by_filter(ThriftHiveMetastore.java:1399)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_by_filter(ThriftHiveMetastore.java:1383)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByFilter(HiveMetaStoreClient.java:680)
at
org.apache.hcatalog.mapreduce.InitializeInput.getSerializedHcatKeyJobInfo(InitializeInput.java:100)
at
org.apache.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:76)
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:40)
... 21 more
================================================================================
--Travis
On Tue, Mar 13, 2012 at 11:28 AM, Ashutosh Chauhan <[email protected]>wrote:
> Hey Travis,
>
> You should never get an exception. Difference in behavior will be whether
> filter got pushed from Pig into HCat or not, but that should not affect
> correctness.
> Can you paste the stack trace that you are getting?
>
> Ashutosh
>
> On Tue, Mar 13, 2012 at 11:14, Travis Crawford <[email protected]
> >wrote:
>
> > Hey HCat gurus -
> >
> > I'm having an issue getting range filters working and am curious if what
> > I'm trying to do makes sense. When filtering for explicit partitions
> (even
> > multiple explicit partitions) things work as expected. However, when
> > requesting a range it fails with a TTransportException.
> >
> > data = LOAD 'db.table' USING org.apache.hcatalog.pig.HCatLoader();
> >
> > -- filter statements that work
> > f = FILTER data BY part_dt == '20120313T000000Z';
> > f = FILTER data BY part_dt == '20120313T000000Z'
> > OR part_dt == '20120313T010000Z';
> >
> > -- filter statements that do not work
> > f = FILTER data BY part_dt >= '20120313T000000Z';
> > f = FILTER data BY part_dt >= '20120313T000000Z'
> > AND part_dt < '20120313T010000Z';
> >
> > When things are working correctly would you expect all four of these
> filter
> > statements to be valid?
> >
> > Thanks!
> > Travis
> >
>
