For ref, I just created a JIRA so people can track the progress/contribute to the progress of HiveIO.
https://issues.apache.org/jira/browse/BEAM-1158 On Wed, Dec 7, 2016 at 5:39 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Yes that's the first idea ;) > > Regards > JB > > On Dec 7, 2016, 17:27, at 17:27, Vinoth Chandar <vin...@uber.com> wrote: > >Interesting. So all the planning & execution is done by Hive, and Beam > >will > >process the results of the query? > > > >On Wed, Dec 7, 2016 at 8:24 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > >wrote: > > > >> Hi > >> > >> The HiveIO will directly use the native API and HiveQL. That's the > >plan on > >> which we are working right now. > >> > >> Regards > >> JB > >> > >> On Dec 7, 2016, 17:18, at 17:18, Vinoth Chandar <vin...@uber.com> > >wrote: > >> >Hi, > >> > > >> >I am not looking for a way to actually execute the query on Hive. I > >> >would > >> >like to do something similar to Spark SQL/HiveContext, but with > >Beam. > >> >Just > >> >have a HiveIO that reads metadata from Hive metastore, and then > >later > >> >use a > >> >Spark runner to execute the query. So, HiveJDBC is not an option I > >> >would > >> >like to pursue. Thanks for the pointer, though! > >> > > >> >And does the HiveIO that is being planned, work similarly as above? > >> > > >> > > >> >Thanks > >> >Vinoth > >> > > >> > > >> > > >> >On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> > >wrote: > >> > > >> >> Hello, > >> >> > >> >> If you really need to read/write via Hive, remember that you can > >use > >> >the > >> >> Hive Jdbc driver, and achieve this with Beam using the JdbcIO > >(this > >> >is > >> >> probably less efficient for the streaming case but still a valid > >> >solution). > >> >> > >> >> Ismaël > >> >> > >> >> > >> >> On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> > >> >wrote: > >> >> > >> >> > Great. Thanks! > >> >> > > >> >> > Thanks, > >> >> > Vinoth > >> >> > > >> >> > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré > >> ><j...@nanthrax.net> > >> >> > wrote: > >> >> > > > >> >> > > Hi, > >> >> > > > >> >> > > Ismaël and I started HiveIO. > >> >> > > > >> >> > > I have several IOs ready to propose as PR, but, in order to > >limit > >> >the > >> >> > number of open PRs, I would like to merge the pending ones. > >> >> > > > >> >> > > I will let you know when the branches/PRs will be available. > >> >> > > > >> >> > > Regards > >> >> > > JB > >> >> > > > >> >> > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: > >> >> > >> Hi guys, > >> >> > >> > >> >> > >> Saw a post around HiveIO on the users list with a PR > >followup. I > >> >am > >> >> > >> interested in this too and can pitch in on developement and > >> >testing.. > >> >> > >> > >> >> > >> Who & where is this work happening? > >> >> > >> > >> >> > >> Thanks > >> >> > >> VInoth > >> >> > >> > >> >> > > > >> >> > > -- > >> >> > > Jean-Baptiste Onofré > >> >> > > jbono...@apache.org > >> >> > > http://blog.nanthrax.net > >> >> > > Talend - http://www.talend.com > >> >> > > >> >> > >> >
