Re: Drill selects column with the same name of a different table

[Divya Gehlot]

You dont have to go ding into Drill config just run the select * query on
the joined data instead of selecting individual column.


Thanks,
Divya

On 22 August 2017 at 19:18, Hielke Hoeve  wrote:

> Hi everyone,
>
> Sorry for not responding but I am not receiving your replies in my
> mailbox. Luckily I have viewed the Archive on the website. Thanks for all
> your replies. I will get back to you asap, I don’t have full access to all
> Drill config so need someone else to do it for me :)
>
> Regards,
>
> Hielke
>
>
>
> > On 21 Aug 2017, at 16:04, Hielke Hoeve  wrote:
> >
> > Hi everyone. I just started using Drill/Zeppelin and I’m facing a
> strange problem. I have a PostgreSQL database linked to a Drill instance.
> Whenever I am trying to join 2 tables which both have a column name and
> whenever I want to select this name Drill selects the wrong name column.
> What am I doing wrong?
> >
> > Given the following 2 tables:
> >
> > Department
> > | id | name |
> > ||--|
> > | 1  |  A   |
> > | 2  |  B   |
> >
> > Employee
> > | id | name | dept | salary |
> > ||--|--||
> > | 1  |  U   |  1   |  100   |
> > | 2  |  V   |  1   |   75   |
> > | 3  |  W   |  1   |  120   |
> > | 4  |  X   |  2   |   95   |
> > | 5  |  Y   |  2   |  140   |
> > | 6  |  Z   |  2   |   55   |
> >
> > Running
> >
> > select employee.name, employee.salary
> > from employee
> > inner join department on employee.dept = department.id
> > where department.name = 'A'
> >
> > returns
> >
> > | name | salary |
> > |--||
> > |  A   |  100   |
> > |  A   |  75|
> > |  A   |  120   |
> >
> > Running
> >
> > select dept.name, employee.salary
> > from employee
> > inner join department on employee.dept = department.id
> > where department.name = 'A'
> >
> > returns
> >
> > | name | salary |
> > |--||
> > | null |  100   |
> > | null |  75|
> > | null |  120   |
> >
> >
> > What does work, but seems very silly to me, is:
> >
> > select dept.name, employee.salary
> > from employee
> > inner join (select id, name as deptname from department) as department
> on employee.dept = department.id
> > where department.deptname = 'A'
> >
> >
>
>

Re: Drills' Hbase storage plugin

[Sorabh Hamirwasia]

Hi Ascot,

When Drill's HBase storage plugin creates a HBase connection it uses the 
storage plugin configuration to get all the required configs. The way storage 
plugin config is formed is based on code I shared with you below or you can 
look in [1]. If you look into HBaseConfiguration.create() then it first loads 
all the HBase related configurations provided as (hbase-site.xml / 
hbase-default.xml) and then override those with Drill's Storage Plugin configs. 
So the reason of suggesting hbase-site.xml with all proper configuration in 
classpath was that the security related configurations will be picked up 
properly from that file and other configuration will be picked from Storage 
Plugin configuration and hence it should work. Hope it helps.


[1]: 
https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePluginConfig.java#L91


Thanks,
Sorabh


From: Ascot Moss 
Sent: Monday, August 21, 2017 12:50:59 AM
To: user@drill.apache.org
Subject: Re: Drills' Hbase storage plugin

Dear Sorabh,

About putting a hbase-site.xml in classpath, can you please explain a bit
more about this point?

Regards

On Mon, Aug 21, 2017 at 3:38 PM, Ascot Moss  wrote:

> Dear Sorabh,
>
> Your reply helps a lot!!
>
> I have copied hbase-site.xml to $DRILL_HOME/conf,
> restarted drillbit,  ran below:
>
> kinit
> sqlline -n "jdbc:drill:drillbit=n1.testserver:31010;auth=kerberos" -n "
> u1@MY_DOMAIN.COM"
>
> use hbase;
> +--+---+
> | ok|   summary|
> +--+---+
> | true  |   Default schema chnaged to [hbase]  |
> +--+---+
>
>
> show tables;
> (no result, took long time)  <=== any idea how to solve it?
>
>
> Regards
>
>
> On Tue, Aug 15, 2017 at 10:14 AM, Sorabh Hamirwasia 
> wrote:
>
>> Hi Ascot,
>>
>> I am not sure if Kerberos between Hbase storage plugin and Hbase server
>> has been tried before or not. Link in [1] talks about Kerberos setup
>> between Drill Client and Drillbit. However looking into the code for Hbase
>> and link at [2], it looks like if you can put a hbase-site.xml in classpath
>> with corresponding setting, then it should work, since internally Drill's
>> hbase storage plugin creates a HBaseConfiguration instance which loads this
>> file in class-path and set corresponding settings for authentication. You
>> need to create TGT for Drill process user (by running kinit, also
>> information provided in link [2]) which will be running as HBase client to
>> the server.
>>
>>
>> HBaseConnectionKey::getHBaseConf() {
>>
>>   return storeConfig.getHBaseConf();
>> }
>>
>>
>> Under HBaseStoragePluginConfig you can find this is how configuration is
>> created:
>>
>>
>> public Configuration getHBaseConf() {
>>   if (hbaseConf == null) {
>> hbaseConf = HBaseConfiguration.create();
>> ...
>>
>> ...
>>
>>   }
>>
>>   ...
>>
>>   ...
>>
>> }
>>
>>
>>
>> [1]: https://drill.apache.org/docs/configuring-kerberos-authentication/<
>> https://drill.apache.org/docs/configuring-kerberos-authentication/><
>> https://drill.apache.org/docs/configuring-kerberos-authentication/>
>> [2]: http://hbase.apache.org/0.94/book/security.html
>>
>>
>> Thanks,
>> Sorabh
>>
>

Re: Avro - Let's talk Avro again

[John Omernik]

This is all great news. Seeing movement, and seeing it articulated, at a
higher level, what has led to the discrepancies in Avro is also very
helpful (thanks Paul).

On Sat, Aug 19, 2017 at 6:29 PM, Stefán Baxter 
wrote:

> Thank you Saurabh,
>
> I was not really expecting a constructive reply to my previous email, was
> appreciated.
>
> I guess some old frustration got the better of me.
>
> All the best,
>  -Stefan
>
> On Fri, Aug 18, 2017 at 10:55 PM, Saurabh Mahapatra <
> saurabhmahapatr...@gmail.com> wrote:
>
> > Thank you for this candid feedback, Stefan. The fact that you even
> decided
> > to write an email offering this feedback despite moving away from Drill
> > just suggests to me that you are still a supporter. We need all the help
> > that we can get from every member in this community to make Drill provide
> > value to all users that include you.
> >
> > I am new to the community but I have looked at your emails where your
> past
> > attempts to doing this have not taken you anywhere. We have to change
> that.
> >
> > We cannot undo the past as far as addressing your needs are concerned
> but I
> > want to assure you that we are bringing reform to the community in
> general.
> > The stakeholders who are impacted by Drill have increased beyond the
> small
> > group that existed a couple of years ago. So be rest assured that you
> have
> > a voice here.
> >
> > I think the biggest challenge we have in the community is that there are
> > users who could get a lot of value if some work was done to support
> > integrations. I know for sure that there are many developers who would
> love
> > to participate in this community and do the work for a modest fee. It
> helps
> > them get interested in the project, helps them provide support beyond
> just
> > the open source aspect and also helps users such as you to get the value
> > that you need where you need it.
> >
> > Please let me know if you would be willing to pursue that route.
> >
> > On the Avro front, I do hear a lot of users asking for it but I hear a
> lot
> > more requests on Parquet. Plus, there are core issues in Drill that needs
> > to be addressed first. The community is definitely trying to prioritize
> > given what we have. But we do not have to feel constrained. We can get
> more
> > developers to participate in this and help out. And I am very positive
> > about that approach-I know that I helped a user here to get help on using
> > Apache Drill inside a commercial setting where there asks were very
> > specific.
> >
> > Those are my thoughts but please do not give up on us. Your critical
> > feedback may not sound nice to the ears but is exactly the kind of
> feedback
> > that will make this project truly successful.
> >
> > Best,
> > Saurabh
> >
> >
> >
> > I
> >
> > On Fri, Aug 18, 2017 at 1:42 PM, Stefán Baxter <
> ste...@activitystream.com>
> > wrote:
> >
> > > Hi John,
> > >
> > > Love Drill but we no longer use it in production as our main query
> tool.
> > >
> > > I do have a fairly long list of pet peeves but I also have a long list
> of
> > > features that I love and would not want to be without.
> > >
> > > In my opinion it's time for Drill to decide where its commitment lies
> > > regarding evolving schema and ETL elimination and if it wants to be
> > > something more than a cogil in a Hadoop distribution wheel or an effort
> > > some see as a way to their startup stardom.
> > >
> > > There is no denying the great effect it has had and its usefulness
> (Arrow
> > > also making waves now). I am, as I have been, just frustrated by
> > > shortcomings I feel are not addressed because they are addressed else
> > where
> > > (where the true loyalties lie)
> > >
> > > I can name a few (I have not upgraded to 1.11):
> > >- Empty values still default to double for partial/segment lists
> which
> > > triggers all sorts of problems  (no attempt is made to convert values
> to
> > > lowest common denominator (string))
> > >- Two NullableX values both containing nothing (Null) still produce
> > > schema change errors instead of waiting for a type to become apparent
> > >- Syntax error reporting is terrible
> > >- Schema change reporting is almost absent
> > >- Avro schema is fixed/strict even though text formats support
> > > evolving/variable schema (With all sorts of side effects)
> > >- Avro still does not support dirN
> > >
> > > and so many more things (not to mention the politics and the defensive
> > > attitude when trying to address shortcomings).
> > >
> > > My only regret here is that I never had proper resources to contribute
> a
> > > fix to some of these.
> > >
> > > All the best,
> > >  -Stefán
> > >
> > > On Thu, Aug 17, 2017 at 2:20 PM, Charles Givre 
> wrote:
> > >
> > > > I’m not an Avro user, but I’d definitely vote for improving this.
> > > > — C
> > > >
> > > > > On Aug 17, 2017, at 10:17, John Omernik  wrote:
> > > > >
> > > > 

Re: Drill selects column with the same name of a different table

[Julian Hyde]

You are not receiving replies because you have not subscribed to the user@drill 
list. Please do that. I moderated your message through last time and this time, 
which is why there was a delay in the messages appearing on the list. 

On 2017-08-22 04:18, Hielke Hoeve  wrote: 
> Hi everyone,
> 
> Sorry for not responding but I am not receiving your replies in my mailbox. 
> Luckily I have viewed the Archive on the website. Thanks for all your 
> replies. I will get back to you asap, I don’t have full access to all Drill 
> config so need someone else to do it for me :)
> 
> Regards,
> 
> Hielke

Re: Drill selects column with the same name of a different table

[Hielke Hoeve]

Hi everyone,

Sorry for not responding but I am not receiving your replies in my mailbox. 
Luckily I have viewed the Archive on the website. Thanks for all your replies. 
I will get back to you asap, I don’t have full access to all Drill config so 
need someone else to do it for me :)

Regards,

Hielke



> On 21 Aug 2017, at 16:04, Hielke Hoeve  wrote:
> 
> Hi everyone. I just started using Drill/Zeppelin and I’m facing a strange 
> problem. I have a PostgreSQL database linked to a Drill instance. Whenever I 
> am trying to join 2 tables which both have a column name and whenever I want 
> to select this name Drill selects the wrong name column. What am I doing 
> wrong?
> 
> Given the following 2 tables:
> 
> Department
> | id | name |
> ||--|
> | 1  |  A   |
> | 2  |  B   |
> 
> Employee
> | id | name | dept | salary |
> ||--|--||
> | 1  |  U   |  1   |  100   |
> | 2  |  V   |  1   |   75   |
> | 3  |  W   |  1   |  120   |
> | 4  |  X   |  2   |   95   |
> | 5  |  Y   |  2   |  140   |
> | 6  |  Z   |  2   |   55   |
> 
> Running
> 
> select employee.name, employee.salary
> from employee
> inner join department on employee.dept = department.id
> where department.name = 'A'
> 
> returns
> 
> | name | salary |
> |--||
> |  A   |  100   |
> |  A   |  75|
> |  A   |  120   |
> 
> Running
> 
> select dept.name, employee.salary
> from employee
> inner join department on employee.dept = department.id
> where department.name = 'A'
> 
> returns
> 
> | name | salary |
> |--||
> | null |  100   |
> | null |  75|
> | null |  120   |
> 
> 
> What does work, but seems very silly to me, is:
> 
> select dept.name, employee.salary
> from employee
> inner join (select id, name as deptname from department) as department on 
> employee.dept = department.id
> where department.deptname = 'A'
> 
>