Had a discussion with Raghu/Zheng offline. Basically, genFileSinkPlan is not passing the type information - that will be added, and then LazySerDe will support serializing into json even if the types are provided
From: Ashish Thusoo [mailto:[email protected]] Sent: Monday, June 29, 2009 11:06 AM To: [email protected] Subject: RE: getting the field types of a query result Not sure what the issue is here, but SemanticAnalyzer does do inferencing of types including that for udfs. I think this is not exposed out through an API, but that can be easily added. Ashish ________________________________ From: Prasad Chakka [mailto:[email protected]] Sent: Monday, June 29, 2009 10:47 AM To: [email protected] Subject: Re: getting the field types of a query result SemanticAnalyzer should set the correct schema of the output result during compilation of the query. Doing the way suggested below is not the right way. If SemanticAnalyzer is not doing this correct thing then we should fix it. ________________________________ From: David Lerman <[email protected]> Reply-To: <[email protected]> Date: Mon, 29 Jun 2009 08:34:25 -0700 To: <[email protected]> Subject: Re: getting the field types of a query result If I'm following's Min's Jira, the challenge is that I would need to parse the query, find the selected fields and look up their type via metaserver, then look at all the UDF's to determine their output type given the inputs -- which is a bit redundant since we already did all that work in executing the query. Min, have you settled on an approach for this in the JDBC driver? > From: He Yongqiang <[email protected]> > Date: Mon, 29 Jun 2009 04:33:59 -0700 > To: <[email protected]> > Subject: Re: getting the field types of a query result > > If I understanding correctly, I think Prasad means is that the type info of > each column is stored in Hive metadata. And you can fetch that information by > HiveServer or JDBC client (and you need to set up a remote hive metaserver). > > Yongqiang > >On 09-6-29 下午6:42, "Min Zhou" <[email protected]> wrote: >> Hi, >> >> I've came across the same problem when developing jdbc for >> hive(https://issues.apache.org/jira/browse/HIVE-576). it had nothing to do >> with HiveServer and jdbc. I thought currently there is no? good way solving >> it. SenmanticAnalyzer needs to get the result's type returned by udf/udaf for >> building the schema you mentioned. we should also consider the user defined >> type . >> >> Regards, >> Min On 6/29/09 11:27 AM, "[email protected]" <[email protected]> wrote: > > > hive-user Digest of: get > > Topics (messages 947 through 950): > > getting the field types of a query result > 947 by: David Lerman > 948 by: Prasad Chakka > 949 by: Min Zhou > 950 by: He Yongqiang > > Administrivia: > > > --- Administrative commands for the hive-user list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > <[email protected]> > > To remove your address from the list, send a message to: > <[email protected]> > > Send mail to the following for info and FAQ for this list: > <[email protected]> > <[email protected]> > > Similar addresses exist for the digest list: > <[email protected]> > <[email protected]> > > To get messages 123 through 145 (a maximum of 100 per request), mail: > <[email protected]> > > To get an index with subject and author for messages 123-456 , mail: > <[email protected]> > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > <[email protected]> > > The messages should contain one line or word of text to avoid being > treated as s...@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "[email protected]", just add a hyphen and your > address (with '=' instead of '@') after the command word: > <[email protected]> > > To stop subscription for this address, mail: > <[email protected]> > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > [email protected]. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: <[email protected]> > Received: (qmail 3099 invoked by uid 99); 29 Jun 2009 15:27:40 -0000 > Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) > by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jun 2009 15:27:40 +0000 > X-ASF-Spam-Status: No, hits=-0.0 required=10.0 > tests=SPF_PASS > X-Spam-Check-By: apache.org > Received-SPF: pass (nike.apache.org: local policy) > Received: from [8.8.14.102] (HELO smtp001.evlta.videoegg.com) (8.8.14.102) > by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jun 2009 15:27:31 +0000 > X-IronPort-AV: E=Sophos;i="4.42,309,1243839600"; > d="scan'208";a="2277316" > Received: from troca1.evlta.ad.videoegg.com ([10.11.12.11]) > by smtp002.evlta.videoegg.com with ESMTP; 29 Jun 2009 08:27:10 -0700 > Received: from troca1.evlta.ad.videoegg.com ([10.11.12.11]) by > troca1.evlta.ad.videoegg.com ([10.11.12.11]) with mapi; Mon, 29 Jun 2009 > 08:25:32 -0700 > From: David Lerman <[email protected]> > To: "[email protected]" <[email protected]> > Date: Mon, 29 Jun 2009 08:27:07 -0700 > Subject: <no subject> > Thread-Topic: <no subject> > Thread-Index: Acn4zg/ffeKshPhRWEmMgyYFzoSarw== > Message-ID: > <c66e558b.11d9f%[email protected]<c66e558b.11d9f%[email protected]>> > Accept-Language: en-US > Content-Language: en > X-MS-Has-Attach: > X-MS-TNEF-Correlator: > acceptlanguage: en-US > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > MIME-Version: 1.0 > X-Virus-Checked: Checked by ClamAV on apache.org > > > > > ---------------------------------------------------------------------- >
