Thanx. Looks like Hcatalog could work. But - is the an example with ‘SELECT’ query?
JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam. From: [gmail.com] [email protected] <[email protected]> Sent: Tuesday, February 11, 2020 10:26 AM To: [email protected] Subject: Re: Apache Beam with Hive While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO. https://beam.apache.org/releases/javadoc/2.18.0/org/apache/beam/sdk/io/hcatalog/HCatalogIO.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org_releases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.html&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=H-EFU9Q7yATgV6oX55NU_3j5EMt2X2k-9B3sJSMou9M&s=NsHbYvkYO2ZYHO43DxCjEdgW_eCBBZJZCMsdq_lZK4o&e=> On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <[email protected]<mailto:[email protected]>> wrote: Hi, If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc: https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sdk_io_jdbc_JdbcIO.java-23L82&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=H-EFU9Q7yATgV6oX55NU_3j5EMt2X2k-9B3sJSMou9M&s=OMH2F23vKbWVyjdXOmsA2hg78Fmp-oxskiaqrnprcII&e=> Regards JB Le 11 févr. 2020 à 07:03, Gershi, Noam <[email protected]<mailto:[email protected]>> a écrit : Hi, I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog? Creating tables, inserting data and fetching it… Noam Gershi Software Developer ICG Technology – TLV Lab T: +972 (3) 7405718 <image002.png>
