Re: [Discuss] Creation of database in Hive

[Pratyaksh Sharma]

Ok, that is a valid reason.

On Thu, Nov 7, 2019 at 2:03 AM Bhavani Sudha 
wrote:

> Ah okay. That is a valid concern. Dint think about admin management for
> Hive dbs.
>
> Thanks,
> Sudha
>
> On Wed, Nov 6, 2019 at 12:28 PM Balaji Varadarajan 
> wrote:
>
> > I have a different opinion on this. Usually, in production deployments
> > (atleast whatever I am aware of), database is generally managed at the
> > org/group level.  Privacy policies like ACLs are usually done at database
> > level and would need first level management by admins. With such a setup,
> > its feels safer to let database creation done through separate process
> and
> > let hudi hive sync only  alter/create tables (current setup).
> >
> > Open to hearing other's thoughts.
> >
> > Regards,
> > Balaji.V
> >
> > On Wed, Nov 6, 2019 at 12:01 PM Bhavani Sudha 
> > wrote:
> >
> > > +1 I think we should create db if it does not exist.
> > >
> > > On Tue, Nov 5, 2019 at 11:08 PM Pratyaksh Sharma <
> pratyaks...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > While doing hive sync using HiveSyncTool, we first check if the
> target
> > > > table exists in hive. If not, we try to create it. However in this
> > flow,
> > > if
> > > > the database itself does not exist, we do not create the database
> > before
> > > > creating hive table, which results in exception like below -
> > > >
> > > > org.apache.hive.service.cli.HiveSQLException: Error while compiling
> > > > statement: FAILED: SemanticException [Error 10072]: Database does not
> > > > exist: test_db
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
> > > > at
> > > org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
> > > > at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at javax.security.auth.Subject.doAs(Subject.java:422)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> > > > at com.sun.proxy.$Proxy68.executeStatementAsync(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
> > > > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> > > > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> > > > ... 3 more
> > > > Caused by: org.apache.hadoop.hive.ql.parse.SemanticException:
> Database
> > > does
> > > > not exist: test_db
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2154)
> > > >
> > > >
> > > > So just wanted to discuss if we should try creating database first in
> > > above
> > > > case using query like -
> > > >
> > > > CREATE DATABASE|SCHEMA [IF NOT EXISTS] 
> > > >
> > >
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Pratyaksh Sharma]

Interested.

Timings:
Mon-Fri 6AM-7.30AM PST

On Thu, Nov 7, 2019 at 11:33 AM Gurudatt Kulkarni 
wrote:

> Interested.
>
> Mon-Thu  5AM-6:30AM PST
> Mon-Thu  9PM-10:30PM PST
>
> These timings work for me.
>
>
> On Thu, Nov 7, 2019 at 10:20 AM Gary Li  wrote:
>
> > Interested.
> > Mon-Thu 8 PM-11 PM PST.
> > It's very difficult to cover America, Europe, and Asia in the same
> meeting.
> > Maybe we can have US&EU and US&CN two sessions and make them biweekly?
> >
> > On Wed, Nov 6, 2019 at 7:12 PM Taher Koitawala 
> wrote:
> >
> > > Hi All,
> > >Mon-Thu  5AM-6:30AM PST
> > >Mon-Thu  9PM-10:30PM PST
> > >
> > > Works for me
> > >
> > > On Thu, Nov 7, 2019, 7:26 AM Nishith  wrote:
> > >
> > > > Following times work for me
> > > >
> > > > Evening : Mon-Thu, 9pm - 1am
> > > >
> > > > Unfortunately, can’t do mornings.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Nov 6, 2019, at 4:51 PM, Y. Ethan Guo  >
> > > > wrote:
> > > > >
> > > > > I'm interested in attending each weekly meeting.  My preferred
> times:
> > > > >
> > > > > Morning: Wed, Fri, 5AM - 7:30AM PT
> > > > > Evening: Mon - Thu, 8PM - 11PM PT
> > > > >
> > > > >
> > > > > On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan <
> > vbal...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Thanks Sudha. The following times work for me :
> > > > >>
> > > > >> Mon, Tue, Thursday - 9 p.m to 12 a.m PST
> > > > >> Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>> On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar <
> vin...@apache.org>
> > > > wrote:
> > > > >>>
> > > > >>> Interested.
> > > > >>>
> > > > >>> Mon-Thu  5AM-6:30AM PST
> > > > >>> Mon-Thu  9PM-10:30PM PST
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha <
> > > bhavanisud...@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > >  Hello all,
> > > > 
> > > >  Currently the weekly sync meeting is scheduled to run on
> Tuesdays
> > > from
> > > > >>> 9pm
> > > >  PST to 10 pm PST. Given our users are from multiple time zones,
> we
> > > can
> > > > >>> try
> > > >  to see if there is any overlapping time that works best. Please
> > > chime
> > > > >> in
> > > > >>> on
> > > >  what would be a suitable time for you if you are interested in
> > > > >> attending
> > > >  the weekly meetings.
> > > > 
> > > >  Thanks,
> > > >  Sudha
> > > > 
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Gurudatt Kulkarni]

Interested.

Mon-Thu  5AM-6:30AM PST
Mon-Thu  9PM-10:30PM PST

These timings work for me.


On Thu, Nov 7, 2019 at 10:20 AM Gary Li  wrote:

> Interested.
> Mon-Thu 8 PM-11 PM PST.
> It's very difficult to cover America, Europe, and Asia in the same meeting.
> Maybe we can have US&EU and US&CN two sessions and make them biweekly?
>
> On Wed, Nov 6, 2019 at 7:12 PM Taher Koitawala  wrote:
>
> > Hi All,
> >Mon-Thu  5AM-6:30AM PST
> >Mon-Thu  9PM-10:30PM PST
> >
> > Works for me
> >
> > On Thu, Nov 7, 2019, 7:26 AM Nishith  wrote:
> >
> > > Following times work for me
> > >
> > > Evening : Mon-Thu, 9pm - 1am
> > >
> > > Unfortunately, can’t do mornings.
> > >
> > > Sent from my iPhone
> > >
> > > > On Nov 6, 2019, at 4:51 PM, Y. Ethan Guo 
> > > wrote:
> > > >
> > > > I'm interested in attending each weekly meeting.  My preferred times:
> > > >
> > > > Morning: Wed, Fri, 5AM - 7:30AM PT
> > > > Evening: Mon - Thu, 8PM - 11PM PT
> > > >
> > > >
> > > > On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan <
> vbal...@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks Sudha. The following times work for me :
> > > >>
> > > >> Mon, Tue, Thursday - 9 p.m to 12 a.m PST
> > > >> Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar 
> > > wrote:
> > > >>>
> > > >>> Interested.
> > > >>>
> > > >>> Mon-Thu  5AM-6:30AM PST
> > > >>> Mon-Thu  9PM-10:30PM PST
> > > >>>
> > > >>>
> > > >>> On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha <
> > bhavanisud...@gmail.com
> > > >
> > > >>> wrote:
> > > >>>
> > >  Hello all,
> > > 
> > >  Currently the weekly sync meeting is scheduled to run on Tuesdays
> > from
> > > >>> 9pm
> > >  PST to 10 pm PST. Given our users are from multiple time zones, we
> > can
> > > >>> try
> > >  to see if there is any overlapping time that works best. Please
> > chime
> > > >> in
> > > >>> on
> > >  what would be a suitable time for you if you are interested in
> > > >> attending
> > >  the weekly meetings.
> > > 
> > >  Thanks,
> > >  Sudha
> > > 
> > > >>>
> > > >>
> > >
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Gary Li]

Interested.
Mon-Thu 8 PM-11 PM PST.
It's very difficult to cover America, Europe, and Asia in the same meeting.
Maybe we can have US&EU and US&CN two sessions and make them biweekly?

On Wed, Nov 6, 2019 at 7:12 PM Taher Koitawala  wrote:

> Hi All,
>Mon-Thu  5AM-6:30AM PST
>Mon-Thu  9PM-10:30PM PST
>
> Works for me
>
> On Thu, Nov 7, 2019, 7:26 AM Nishith  wrote:
>
> > Following times work for me
> >
> > Evening : Mon-Thu, 9pm - 1am
> >
> > Unfortunately, can’t do mornings.
> >
> > Sent from my iPhone
> >
> > > On Nov 6, 2019, at 4:51 PM, Y. Ethan Guo 
> > wrote:
> > >
> > > I'm interested in attending each weekly meeting.  My preferred times:
> > >
> > > Morning: Wed, Fri, 5AM - 7:30AM PT
> > > Evening: Mon - Thu, 8PM - 11PM PT
> > >
> > >
> > > On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan 
> > > wrote:
> > >
> > >> Thanks Sudha. The following times work for me :
> > >>
> > >> Mon, Tue, Thursday - 9 p.m to 12 a.m PST
> > >> Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
> > >>
> > >>
> > >>
> > >>
> > >>> On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar 
> > wrote:
> > >>>
> > >>> Interested.
> > >>>
> > >>> Mon-Thu  5AM-6:30AM PST
> > >>> Mon-Thu  9PM-10:30PM PST
> > >>>
> > >>>
> > >>> On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha <
> bhavanisud...@gmail.com
> > >
> > >>> wrote:
> > >>>
> >  Hello all,
> > 
> >  Currently the weekly sync meeting is scheduled to run on Tuesdays
> from
> > >>> 9pm
> >  PST to 10 pm PST. Given our users are from multiple time zones, we
> can
> > >>> try
> >  to see if there is any overlapping time that works best. Please
> chime
> > >> in
> > >>> on
> >  what would be a suitable time for you if you are interested in
> > >> attending
> >  the weekly meetings.
> > 
> >  Thanks,
> >  Sudha
> > 
> > >>>
> > >>
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Taher Koitawala]

Hi All,
   Mon-Thu  5AM-6:30AM PST
   Mon-Thu  9PM-10:30PM PST

Works for me

On Thu, Nov 7, 2019, 7:26 AM Nishith  wrote:

> Following times work for me
>
> Evening : Mon-Thu, 9pm - 1am
>
> Unfortunately, can’t do mornings.
>
> Sent from my iPhone
>
> > On Nov 6, 2019, at 4:51 PM, Y. Ethan Guo 
> wrote:
> >
> > I'm interested in attending each weekly meeting.  My preferred times:
> >
> > Morning: Wed, Fri, 5AM - 7:30AM PT
> > Evening: Mon - Thu, 8PM - 11PM PT
> >
> >
> > On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan 
> > wrote:
> >
> >> Thanks Sudha. The following times work for me :
> >>
> >> Mon, Tue, Thursday - 9 p.m to 12 a.m PST
> >> Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
> >>
> >>
> >>
> >>
> >>> On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar 
> wrote:
> >>>
> >>> Interested.
> >>>
> >>> Mon-Thu  5AM-6:30AM PST
> >>> Mon-Thu  9PM-10:30PM PST
> >>>
> >>>
> >>> On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha  >
> >>> wrote:
> >>>
>  Hello all,
> 
>  Currently the weekly sync meeting is scheduled to run on Tuesdays from
> >>> 9pm
>  PST to 10 pm PST. Given our users are from multiple time zones, we can
> >>> try
>  to see if there is any overlapping time that works best. Please chime
> >> in
> >>> on
>  what would be a suitable time for you if you are interested in
> >> attending
>  the weekly meetings.
> 
>  Thanks,
>  Sudha
> 
> >>>
> >>
>

Re: [Discuss] Convenient time for weekly sync meeting

[Nishith]

Following times work for me 

Evening : Mon-Thu, 9pm - 1am

Unfortunately, can’t do mornings.

Sent from my iPhone

> On Nov 6, 2019, at 4:51 PM, Y. Ethan Guo  wrote:
> 
> I'm interested in attending each weekly meeting.  My preferred times:
> 
> Morning: Wed, Fri, 5AM - 7:30AM PT
> Evening: Mon - Thu, 8PM - 11PM PT
> 
> 
> On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan 
> wrote:
> 
>> Thanks Sudha. The following times work for me :
>> 
>> Mon, Tue, Thursday - 9 p.m to 12 a.m PST
>> Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
>> 
>> 
>> 
>> 
>>> On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar  wrote:
>>> 
>>> Interested.
>>> 
>>> Mon-Thu  5AM-6:30AM PST
>>> Mon-Thu  9PM-10:30PM PST
>>> 
>>> 
>>> On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha 
>>> wrote:
>>> 
 Hello all,
 
 Currently the weekly sync meeting is scheduled to run on Tuesdays from
>>> 9pm
 PST to 10 pm PST. Given our users are from multiple time zones, we can
>>> try
 to see if there is any overlapping time that works best. Please chime
>> in
>>> on
 what would be a suitable time for you if you are interested in
>> attending
 the weekly meetings.
 
 Thanks,
 Sudha
 
>>> 
>> 

Re: [Discuss] Convenient time for weekly sync meeting

[leesf]

Thanks Sudha. Interested.

Tue - Thu, 8:30PM - 10:00PM PST
Wed - Fri, 3:00AM - 4:30AM PST


Y. Ethan Guo  于2019年11月7日周四 上午8:52写道：

> I'm interested in attending each weekly meeting.  My preferred times:
>
> Morning: Wed, Fri, 5AM - 7:30AM PT
> Evening: Mon - Thu, 8PM - 11PM PT
>
>
> On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan 
> wrote:
>
> > Thanks Sudha. The following times work for me :
> >
> > Mon, Tue, Thursday - 9 p.m to 12 a.m PST
> > Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
> >
> >
> >
> >
> > On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar 
> wrote:
> >
> > > Interested.
> > >
> > > Mon-Thu  5AM-6:30AM PST
> > > Mon-Thu  9PM-10:30PM PST
> > >
> > >
> > > On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha  >
> > > wrote:
> > >
> > > > Hello all,
> > > >
> > > > Currently the weekly sync meeting is scheduled to run on Tuesdays
> from
> > > 9pm
> > > > PST to 10 pm PST. Given our users are from multiple time zones, we
> can
> > > try
> > > > to see if there is any overlapping time that works best. Please chime
> > in
> > > on
> > > > what would be a suitable time for you if you are interested in
> > attending
> > > > the weekly meetings.
> > > >
> > > > Thanks,
> > > > Sudha
> > > >
> > >
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Y. Ethan Guo]

I'm interested in attending each weekly meeting.  My preferred times:

Morning: Wed, Fri, 5AM - 7:30AM PT
Evening: Mon - Thu, 8PM - 11PM PT


On Wed, Nov 6, 2019 at 3:12 PM Balaji Varadarajan 
wrote:

> Thanks Sudha. The following times work for me :
>
> Mon, Tue, Thursday - 9 p.m to 12 a.m PST
> Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST
>
>
>
>
> On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar  wrote:
>
> > Interested.
> >
> > Mon-Thu  5AM-6:30AM PST
> > Mon-Thu  9PM-10:30PM PST
> >
> >
> > On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha 
> > wrote:
> >
> > > Hello all,
> > >
> > > Currently the weekly sync meeting is scheduled to run on Tuesdays from
> > 9pm
> > > PST to 10 pm PST. Given our users are from multiple time zones, we can
> > try
> > > to see if there is any overlapping time that works best. Please chime
> in
> > on
> > > what would be a suitable time for you if you are interested in
> attending
> > > the weekly meetings.
> > >
> > > Thanks,
> > > Sudha
> > >
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Balaji Varadarajan]

Thanks Sudha. The following times work for me :

Mon, Tue, Thursday - 9 p.m to 12 a.m PST
Wed - 5:00 to 6:00 am and 9:30 p.m to 12 a.m PST




On Wed, Nov 6, 2019 at 12:31 PM Vinoth Chandar  wrote:

> Interested.
>
> Mon-Thu  5AM-6:30AM PST
> Mon-Thu  9PM-10:30PM PST
>
>
> On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha 
> wrote:
>
> > Hello all,
> >
> > Currently the weekly sync meeting is scheduled to run on Tuesdays from
> 9pm
> > PST to 10 pm PST. Given our users are from multiple time zones, we can
> try
> > to see if there is any overlapping time that works best. Please chime in
> on
> > what would be a suitable time for you if you are interested in attending
> > the weekly meetings.
> >
> > Thanks,
> > Sudha
> >
>

Re: [Discuss] Creation of database in Hive

[Bhavani Sudha]

Ah okay. That is a valid concern. Dint think about admin management for
Hive dbs.

Thanks,
Sudha

On Wed, Nov 6, 2019 at 12:28 PM Balaji Varadarajan 
wrote:

> I have a different opinion on this. Usually, in production deployments
> (atleast whatever I am aware of), database is generally managed at the
> org/group level.  Privacy policies like ACLs are usually done at database
> level and would need first level management by admins. With such a setup,
> its feels safer to let database creation done through separate process and
> let hudi hive sync only  alter/create tables (current setup).
>
> Open to hearing other's thoughts.
>
> Regards,
> Balaji.V
>
> On Wed, Nov 6, 2019 at 12:01 PM Bhavani Sudha 
> wrote:
>
> > +1 I think we should create db if it does not exist.
> >
> > On Tue, Nov 5, 2019 at 11:08 PM Pratyaksh Sharma 
> > wrote:
> >
> > > Hi,
> > >
> > > While doing hive sync using HiveSyncTool, we first check if the target
> > > table exists in hive. If not, we try to create it. However in this
> flow,
> > if
> > > the database itself does not exist, we do not create the database
> before
> > > creating hive table, which results in exception like below -
> > >
> > > org.apache.hive.service.cli.HiveSQLException: Error while compiling
> > > statement: FAILED: SemanticException [Error 10072]: Database does not
> > > exist: test_db
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
> > > at
> > org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
> > > at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:422)
> > > at
> > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> > > at com.sun.proxy.$Proxy68.executeStatementAsync(Unknown Source)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
> > > at
> > >
> > >
> >
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
> > > at
> > >
> > >
> >
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
> > > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> > > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> > > at
> > >
> > >
> >
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> > > at
> > >
> > >
> >
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> > > ... 3 more
> > > Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Database
> > does
> > > not exist: test_db
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2154)
> > >
> > >
> > > So just wanted to discuss if we should try creating database first in
> > above
> > > case using query like -
> > >
> > > CREATE DATABASE|SCHEMA [IF NOT EXISTS] 
> > >
> >
>

Re: [Discuss] Convenient time for weekly sync meeting

[Vinoth Chandar]

Interested.

Mon-Thu  5AM-6:30AM PST
Mon-Thu  9PM-10:30PM PST


On Wed, Nov 6, 2019 at 12:28 PM Bhavani Sudha 
wrote:

> Hello all,
>
> Currently the weekly sync meeting is scheduled to run on Tuesdays from 9pm
> PST to 10 pm PST. Given our users are from multiple time zones, we can try
> to see if there is any overlapping time that works best. Please chime in on
> what would be a suitable time for you if you are interested in attending
> the weekly meetings.
>
> Thanks,
> Sudha
>

[Discuss] Convenient time for weekly sync meeting

[Bhavani Sudha]

Hello all,

Currently the weekly sync meeting is scheduled to run on Tuesdays from 9pm
PST to 10 pm PST. Given our users are from multiple time zones, we can try
to see if there is any overlapping time that works best. Please chime in on
what would be a suitable time for you if you are interested in attending
the weekly meetings.

Thanks,
Sudha

Re: [Discuss] Creation of database in Hive

[Balaji Varadarajan]

I have a different opinion on this. Usually, in production deployments
(atleast whatever I am aware of), database is generally managed at the
org/group level.  Privacy policies like ACLs are usually done at database
level and would need first level management by admins. With such a setup,
its feels safer to let database creation done through separate process and
let hudi hive sync only  alter/create tables (current setup).

Open to hearing other's thoughts.

Regards,
Balaji.V

On Wed, Nov 6, 2019 at 12:01 PM Bhavani Sudha 
wrote:

> +1 I think we should create db if it does not exist.
>
> On Tue, Nov 5, 2019 at 11:08 PM Pratyaksh Sharma 
> wrote:
>
> > Hi,
> >
> > While doing hive sync using HiveSyncTool, we first check if the target
> > table exists in hive. If not, we try to create it. However in this flow,
> if
> > the database itself does not exist, we do not create the database before
> > creating hive table, which results in exception like below -
> >
> > org.apache.hive.service.cli.HiveSQLException: Error while compiling
> > statement: FAILED: SemanticException [Error 10072]: Database does not
> > exist: test_db
> > at
> >
> >
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> > at
> >
> >
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
> > at
> >
> >
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
> > at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> > at
> >
> >
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575)
> > at
> >
> >
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
> > at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> > at
> >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> > at
> >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> > at
> >
> >
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> > at com.sun.proxy.$Proxy68.executeStatementAsync(Unknown Source)
> > at
> >
> >
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
> > at
> >
> >
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
> > at
> >
> >
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
> > at
> >
> >
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
> > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> > at
> >
> >
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> > at
> >
> >
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> > ... 3 more
> > Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Database
> does
> > not exist: test_db
> > at
> >
> >
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2154)
> >
> >
> > So just wanted to discuss if we should try creating database first in
> above
> > case using query like -
> >
> > CREATE DATABASE|SCHEMA [IF NOT EXISTS] 
> >
>

Re: [Nov 5, 2019] Weekly Sync meeting

[Bhavani Sudha]

Thanks Kabeer for the valuable feedback. Let me start a [DISCUSS] thread
around this to get a sense of interest and time availabilities from the
community.

Thanks,
Sudha



On Wed, Nov 6, 2019 at 3:13 AM Kabeer Ahmed  wrote:

> Sudha, Balaji, and All,
>
> Thank you so much for this initiative. 9pm PST is 5am GMT and quite
> inconvenient for people in Europe. I think a fair way would be to poll
> based on existing users and then decide on a time. Getting a convenient
> time for people in West Coast to far east is always challenging.
> Please let me know if such exercise has been carried out to arrive at the
> time that has been decided.
> Thanks
> Kabeer.
>
> On Nov 6 2019, at 6:14 am, Bhavani Sudha  wrote:
> > Please find the meeting minutes here - 20191105 Weekly Sync Minutes
> > 
> >
> > Thanks,
> > Sudha
> >
>
>

Re: [Discuss] Creation of database in Hive

[Bhavani Sudha]

+1 I think we should create db if it does not exist.

On Tue, Nov 5, 2019 at 11:08 PM Pratyaksh Sharma 
wrote:

> Hi,
>
> While doing hive sync using HiveSyncTool, we first check if the target
> table exists in hive. If not, we try to create it. However in this flow, if
> the database itself does not exist, we do not create the database before
> creating hive table, which results in exception like below -
>
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: SemanticException [Error 10072]: Database does not
> exist: test_db
> at
>
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> at
>
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
> at
>
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> at
>
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575)
> at
>
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
> at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
>
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
>
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
>
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy68.executeStatementAsync(Unknown Source)
> at
>
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
> at
>
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
> at
>
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
> at
>
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
>
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Database does
> not exist: test_db
> at
>
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2154)
>
>
> So just wanted to discuss if we should try creating database first in above
> case using query like -
>
> CREATE DATABASE|SCHEMA [IF NOT EXISTS] 
>

Re: [Nov 5, 2019] Weekly Sync meeting

[Kabeer Ahmed]

Sudha, Balaji, and All,

Thank you so much for this initiative. 9pm PST is 5am GMT and quite 
inconvenient for people in Europe. I think a fair way would be to poll based on 
existing users and then decide on a time. Getting a convenient time for people 
in West Coast to far east is always challenging.
Please let me know if such exercise has been carried out to arrive at the time 
that has been decided.
Thanks
Kabeer.

On Nov 6 2019, at 6:14 am, Bhavani Sudha  wrote:
> Please find the meeting minutes here - 20191105 Weekly Sync Minutes
> 
>
> Thanks,
> Sudha
>