Hi Jorn, Create external table 'mytable' ( mydata string) row format serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as inputformat 'com.hello.world.myinputformat' outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' location '/tmp/mysampledata' TBLPROPERTIES ( 'myinputformat.setting1'='test' );
I'm not using a HiveSerDe at the moment, I'm just looking to parse a multiline log file into logical records before applying SerDe. No exceptions in the log. SELECT * returns correctly and prints out logging statements in the HiveServer2 log, SELECT count (*) does not appear to trigger any of my logging statements to fire into the HiveServer2 log so I don't see anything. Of course, it's not useful as INSERT INTO SELECT from mytable doesn't insert any rows. My theory is that the JobConf object is not populated from the TBLPROPERTIES when a SELECT count(*) is called. If I use a "set" command, SELECT count(*) finds that correctly and my inputformat works as expected. If this is the case, the limitation is when I'm working with more than one table using the same input format but different value for the "myinputformat.setting1" setting. Kind Regards Chris On Mon, 19 Dec 2016 at 17:08 Jörn Franke <jornfra...@gmail.com> wrote: > What is the create table statement? Do you parse the tblproperties in the > HiveSerde? Do you have exceptions in the log? > > > On 19 Dec 2016, at 07:02, Chris Teoh <chris.t...@gmail.com> wrote: > > > > Hi there, > > > > Can anyone confirm whether TBLPROPERTIES in DDLs are ignored by custom > > inputformats in the context of a UDAF? > > > > I've written a custom input format and it works with a SELECT * but when > I > > do anything more like SELECT count(*) it returns 0. > > > > INSERT INTO <table> SELECT * FROM <my custom inputformat table> doesn't > > appear to insert anything into the table. > > > > I found I had to use the "set" commands to make things work rather than > use > > DDL. This doesn't appear to make sense if the setting is specific to the > > table and would cause problems if I was working with more than one table > > and needed different values for the same setting. > > > > Kind regards > > Chris >
