RE: TIMESTAMPADD sql jdbc function execution in drill

[Sudip Mukherjee]

Thanks Khurram.

I am not sure if I am looking at the correct place. Seeing it in calcite,
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/SqlJdbcFunctionCall.java
 

I'll try to check further on the date-time functions.

Thanks,
Sudip

-Original Message-
From: Khurram Faraaz [mailto:kfar...@maprtech.com] 
Sent: 08 February 2016 PM 12:14
To: dev@drill.apache.org
Subject: Re: TIMESTAMPADD sql jdbc function execution in drill

I didn't find it on the documented date time functions here 
https://drill.apache.org/docs/date-time-functions-and-arithmetic/

On Mon, Feb 8, 2016 at 12:09 PM, Sudip Mukherjee 
wrote:

> Hi,
> Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I 
> am getting an error while running the below query
>
> SELECT CAST(EXTRACT(YEAR FROM CAST({fn 
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d 
> '1970-01-01'})} AS DATE)) AS INTEGER) AS 
> `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
> FROM `dfs.data`.`MyFileShare_2214` `myshare` GROUP BY 
> CAST(EXTRACT(YEAR FROM CAST({fn 
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d 
> '1970-01-01'})} AS DATE)) AS INTEGER)
>
>
> VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column 
> 'SQL_TSI_YEAR' not found in any table
>
> Thanks,
> Sudip
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Re: TIMESTAMPADD sql jdbc function execution in drill

[Arina Yelchiyeva]

Hi Sudip,

timestampadd / timestampdiff functions are not currently implemented in
Drill but they are in development.
See Jira https://issues.apache.org/jira/browse/DRILL-3610


Kind regards
Arina

On Mon, Feb 8, 2016 at 9:29 AM Sudip Mukherjee 
wrote:

> Thanks Khurram.
>
> I am not sure if I am looking at the correct place. Seeing it in calcite,
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/SqlJdbcFunctionCall.java
>
> I'll try to check further on the date-time functions.
>
> Thanks,
> Sudip
>
> -Original Message-
> From: Khurram Faraaz [mailto:kfar...@maprtech.com]
> Sent: 08 February 2016 PM 12:14
> To: dev@drill.apache.org
> Subject: Re: TIMESTAMPADD sql jdbc function execution in drill
>
> I didn't find it on the documented date time functions here
> https://drill.apache.org/docs/date-time-functions-and-arithmetic/
>
> On Mon, Feb 8, 2016 at 12:09 PM, Sudip Mukherjee  >
> wrote:
>
> > Hi,
> > Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I
> > am getting an error while running the below query
> >
> > SELECT CAST(EXTRACT(YEAR FROM CAST({fn
> > TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> > '1970-01-01'})} AS DATE)) AS INTEGER) AS
> > `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
> > FROM `dfs.data`.`MyFileShare_2214` `myshare` GROUP BY
> > CAST(EXTRACT(YEAR FROM CAST({fn
> > TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> > '1970-01-01'})} AS DATE)) AS INTEGER)
> >
> >
> > VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column
> > 'SQL_TSI_YEAR' not found in any table
> >
> > Thanks,
> > Sudip
> >
> >
> >
> > ***Legal Disclaimer***
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply
> > email and delete the message. Thank you."
> > **
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **

Re: [VOTE] Release Apache Drill 1.5.0 RC2

[Aman Sinha]

+1
- Downloaded src and built, ran unit tests on my Mac
- Manually ran a few queries against TPC-DS
- Verified partition pruning, metadata caching was working as expected for
these test queries
- Checked query profile in Web UI, checked query cancellation
- Found 1 performance issue with lots of small parquet files ...filed
DRILL-4365 but need confirmation whether it is reproducible for other
folks.  At this point, I am not considering it a blocker due to the fact I
could not reproduce with a more general/bigger dataset.

Aman

On Fri, Feb 5, 2016 at 12:21 PM, Julien Le Dem  wrote:

> +1 (non-binding)
> Built and run the tests on linux (took 27 min)
>
>
>
> On Fri, Feb 5, 2016 at 11:21 AM, Stefán Baxter 
> wrote:
>
> > +1 (non-binding / not a committer)
> >
> >- Built the project on ubuntu/linux
> >- Ran our test suite
> >- Verified that the jdbc driver works and is properly shaded (we had
> >problems with *leakage*)
> >
> > (I ran into a problem reading a snappy zipped parquet file that was
> created
> > with the latest parquet-mr/parquet-avro (1.8.1) but i think that is out
> of
> > scope here and I will create a Jira issue once I have tested it better)
> >
> > Thank you
> >
> > On Fri, Feb 5, 2016 at 6:56 PM, Jason Altekruse <
> altekruseja...@gmail.com>
> > wrote:
> >
> > > Hello all,
> > >
> > > I'd like to propose the third release candidate (rc2) of Apache Drill,
> > > version
> > > 1.5.0. It covers a total of 55 resolved JIRAs [1]. Thanks to everyone
> who
> > > contributed to this release. This release candidate includes a fix for
> > > DRILL-4353, a major stability problem with the Rest API that was
> > identified
> > > during the last vote.
> > >
> > > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted
> > > at
> > > [3]. This release candidate is based on commit
> > > 0a64888ba8d374e94435e2518e81352e677255ad located at [4].
> > >
> > > The vote will be open for the next 96 hours (including an extra day as
> > the
> > > vote is happening over a weekend) ending at 11AM Pacific, February 9th,
> > > 2016.
> > >
> > > [ ] +1
> > > [ ] +0
> > > [ ] -1
> > >
> > > Here's my vote: +1
> > >
> > > Thanks,
> > > Jason
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12332948
> > > [2] http://people.apache.org/~json/apache-drill-1.5.0.rc2/
> > > [3]
> > https://repository.apache.org/content/repositories/orgapachedrill-1026
> > > [4] https://github.com/jaltekruse/incubator-drill/tree/1.5-release-rc2
> > >
> >
>
>
>
> --
> Julien
>

[jira] [Created] (DRILL-4365) Performance with lots of small parquet files

[Aman Sinha (JIRA)]

Aman Sinha created DRILL-4365:
-

 Summary: Performance with lots of small parquet files
 Key: DRILL-4365
 URL: https://issues.apache.org/jira/browse/DRILL-4365
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.5.0
Reporter: Aman Sinha


I am seeing a performance degradation on 1.5.0 compared to 1.4.0 with a query 
over 968 small parquet files where the total # rows is only 1000, so just about 
1 row per file.  The profile shows parquet scan is slower.  With bigger  
tables, I haven't seen the same issue yet (although need confirmation from the 
full performance run).  

Note: this is with default slice_target of 100K so only 1 scan fragment was 
used.   I will attach the dataset to this JIRA if anyone wants to repro. 

On 1.4.0: (with multiple runs):

{noformat}
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+-+
| EXPR$0  |
+-+
| 39  |
+-+
1 row selected (2.544 seconds)
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+-+
| EXPR$0  |
+-+
| 39  |
+-+
1 row selected (2.434 seconds)
{noformat}

On 1.5.0:  (multiple runs):

{noformat}
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+-+
| EXPR$0  |
+-+
| 39  |
+-+
1 row selected (3.851 seconds)
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+-+
| EXPR$0  |
+-+
| 39  |
+-+
1 row selected (3.61 seconds)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4366) case expression referencing columns[..] = literal fails to resolve to tru

[N Campbell (JIRA)]

N Campbell created DRILL-4366:
-

 Summary: case expression referencing columns[..] = literal fails 
to resolve to tru
 Key: DRILL-4366
 URL: https://issues.apache.org/jira/browse/DRILL-4366
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser
Affects Versions: 1.4.0
Reporter: N Campbell


A file (i.e PSV) has one or more columns with the string literal \N. A query 
that projects a conditional value based on testing the columns[N] being equal 
never results in true. Meanwhile comparing the same literal to the result of  a 
substring applied to the same column works. 

SELECT columns[0] as RNUM, columns[1], case when substr(columns[1],1,2)= '\N' 
then 1 else 0 end ,case when columns[1]='\N' then 1 else 0 end  FROM 
`dfs`.`TEXT`.`./TBINT.tbl` LIMIT 100

input file TBINT.tbl

0|\N
1|-1
2|0
3|1
4|10




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4132 Ability to submit simple type of ph...

[yufeldman]

GitHub user yufeldman opened a pull request:

https://github.com/apache/drill/pull/368

DRILL-4132 Ability to submit simple type of physical plan directly to…

… EndPoint DrillBit for execution.

There are multiple changes to achieve this:
1. During physical planning split single plan into multiple based on the 
number of minor fragments of the Leaf Major fragment.
   a. Removing exchange operators during planning
   b. Producing just root fragments (that will be also leaf fragments)
2. Each fragment can be executed against Drillbit it is assigned to, so to 
keep locality
Design document can be found in the JIRA: DRILL-4132

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yufeldman/incubator-drill DRILL-4132

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/368.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #368


commit 2a35967396dd66c3e371973fe8baf14f720ed219
Author: Yuliya Feldman 
Date:   2016-02-04T22:09:21Z

DRILL-4132 Ability to submit simple type of physical plan directly to 
EndPoint DrillBit for execution.
There are multiple changes to achieve this:
1. During physical planning split single plan into multiple based on the 
number of minor fragments of the Leaf Major fragment.
   a. Removing exchange operators during planning
   b. Producing just root fragments (that will be also leaf fragments)
2. Each fragment can be executed against Drillbit it is assigned to, so to 
keep locality
Design document can be found in the JIRA: DRILL-4132




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

TIMESTAMPADD sql jdbc function execution in drill

[Sudip Mukherjee]

Hi,
Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I am 
getting an error while running the below query

SELECT CAST(EXTRACT(YEAR FROM CAST({fn 
TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d '1970-01-01'})} 
AS DATE)) AS INTEGER) AS `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
FROM `dfs.data`.`MyFileShare_2214` `myshare`
GROUP BY CAST(EXTRACT(YEAR FROM CAST({fn 
TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d '1970-01-01'})} 
AS DATE)) AS INTEGER)


VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column 
'SQL_TSI_YEAR' not found in any table

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Re: TIMESTAMPADD sql jdbc function execution in drill

[Khurram Faraaz]

I didn't find it on the documented date time functions here
https://drill.apache.org/docs/date-time-functions-and-arithmetic/

On Mon, Feb 8, 2016 at 12:09 PM, Sudip Mukherjee 
wrote:

> Hi,
> Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I am
> getting an error while running the below query
>
> SELECT CAST(EXTRACT(YEAR FROM CAST({fn
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> '1970-01-01'})} AS DATE)) AS INTEGER) AS
> `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
> FROM `dfs.data`.`MyFileShare_2214` `myshare`
> GROUP BY CAST(EXTRACT(YEAR FROM CAST({fn
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> '1970-01-01'})} AS DATE)) AS INTEGER)
>
>
> VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column
> 'SQL_TSI_YEAR' not found in any table
>
> Thanks,
> Sudip
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **