[GitHub] drill pull request #793: DRILL-4678: Tune metadata by generating a dispatche...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/793#discussion_r107647154
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java
 ---
@@ -14,35 +14,71 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
- 
**/
+ */
 package org.apache.drill.exec.planner.cost;
 
+import org.apache.calcite.rel.SingleRel;
 import org.apache.calcite.rel.core.Aggregate;
 import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
 import org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
 import org.apache.calcite.rel.metadata.RelMdRowCount;
 import org.apache.calcite.rel.metadata.RelMetadataProvider;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
 import org.apache.calcite.util.BuiltInMethod;
 import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.drill.exec.planner.common.DrillLimitRelBase;
 
 public class DrillRelMdRowCount extends RelMdRowCount{
   private static final DrillRelMdRowCount INSTANCE = new 
DrillRelMdRowCount();
 
   public static final RelMetadataProvider SOURCE = 
ReflectiveRelMetadataProvider.reflectiveSource(BuiltInMethod.ROW_COUNT.method, 
INSTANCE);
 
   @Override
-  public Double getRowCount(Aggregate rel) {
+  public Double getRowCount(Aggregate rel, RelMetadataQuery mq) {
 ImmutableBitSet groupKey = ImmutableBitSet.range(rel.getGroupCount());
 
 if (groupKey.isEmpty()) {
   return 1.0;
 } else {
-  return super.getRowCount(rel);
+  return super.getRowCount(rel, mq);
 }
   }
 
   @Override
-  public Double getRowCount(Filter rel) {
-return rel.getRows();
+  public Double getRowCount(Filter rel, RelMetadataQuery mq) {
--- End diff --

In RelMdRowCount.java added overloaded methods, which returns other result 
than getRowCount(RelNode rel, RelMetadataQuery mq) [1] as it was before. 
So in order to preserve previous Drill behavior, I override those methods.
[1] 
https://github.com/Serhii-Harnyk/incubator-calcite/blob/02848c99d75f2d0e00c219f3fa300fc8e7df26df/core/src/main/java/org/apache/calcite/rel/metadata/RelMdRowCount.java#L65


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #793: DRILL-4678: Tune metadata by generating a dispatche...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/793

DRILL-4678: Tune metadata by generating a dispatcher at runtime

Changes for rebasing to Calcite 1.4.0-drill-r20 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-4678

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/793.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #793


commit 794e9537bbe7d25abe3809db5847d898c45c73e1
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2017-03-03T15:24:26Z

DRILL-4678: Tune metadata by generating a dispatcher at runtime




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #748: DRILL-5263: Prevent left NLJoin with non scalar subqueries

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/748
  
Merged into apace master without message _close apache/drill#748_. Closing 
PR manually.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...

[Serhii-Harnyk]

Github user Serhii-Harnyk closed the pull request at:

https://github.com/apache/drill/pull/748


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/748#discussion_r101308389
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/NestedLoopJoinPrule.java
 ---
@@ -49,7 +49,7 @@ protected boolean checkPreconditions(DrillJoinRel join, 
RelNode left, RelNode ri
   PlannerSettings settings) {
 JoinRelType type = join.getJoinType();
 
-if (! (type == JoinRelType.INNER || type == JoinRelType.LEFT)) {
+if (!(type == JoinRelType.INNER || (type == JoinRelType.LEFT && 
JoinUtils.hasScalarSubqueryInput(left, right {
--- End diff --

Yes, you a right, we should return error in this case too. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/748#discussion_r101086894
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/NestedLoopJoinPrule.java
 ---
@@ -49,7 +49,7 @@ protected boolean checkPreconditions(DrillJoinRel join, 
RelNode left, RelNode ri
   PlannerSettings settings) {
 JoinRelType type = join.getJoinType();
 
-if (! (type == JoinRelType.INNER || type == JoinRelType.LEFT)) {
+if (!(type == JoinRelType.INNER || (type == JoinRelType.LEFT && 
JoinUtils.hasScalarSubqueryInput(left, right {
--- End diff --

In the case when table joining with the result of the query with aggregate 
function, Drill creates physical plan with left nested loop join.
It is due to this code line:

https://github.com/mapr/incubator-calcite/blob/DrillCalcite1.4.0/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1078
So to allow execution of such queries without changes in calcite - allow 
left nested loop join for scalar subqueries.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
@amansinha100, could you please review new changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r101024133
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -492,10 +537,18 @@ private void writeDataAllText(MapWriter map, 
FieldSelection selection,
   }
 }
 map.end();
-
   }
 
   /**
+   * Puts copy of field path list to fieldPathWriter map.
+   * @param fieldName
+   */
+  private void putFieldPath(String fieldName, MapWriter map) {
+List fieldPath = Lists.newArrayList(path);
--- End diff --

Thanks, fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/748

DRILL-5263: Prevent left NLJoin with non scalar subqueries



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5263

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/748.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #748


commit fff774ed62bf85f36a0167b1e8c49c1339312131
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2017-02-13T14:30:27Z

DRILL-5263: Prevent left NLJoin with non scalar subqueries




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5263:


 Summary: Prevent left NLJoin with non scalar subqueries
 Key: DRILL-5263
 URL: https://issues.apache.org/jira/browse/DRILL-5263
 Project: Apache Drill
  Issue Type: Bug
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join and non scalar sub-queries. Drill 
should throw error in this case.
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
Squashed all changes into single commit, rebased on master and resolved 
conflicts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/581
  
@paul-rogers could you please review new changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99315319
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJod

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99313550
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJod

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99315665
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJod

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99316546
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,216 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set postgresValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap postgresToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  // tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  // postgres patterns
+  public static final String POSTGRES_FULL_NAME_OF_DAY = "day";
+  public static final String POSTGRES_DAY_OF_YEAR = "ddd";
+  public static final String POSTGRES_DAY_OF_MONTH = "dd";
+  public static final String POSTGRES_DAY_OF_WEEK = "d";
+  public static final String POSTGRES_NAME_OF_MONTH = "month";
+  public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon";
+  public static final String POSTGRES_FULL_ERA_NAME = "ee";
+  public static final String POSTGRES_NAME_OF_DAY = "dy";
+  public static final String POSTGRES_TIME_ZONE_NAME = "tz";
+  public static final String POSTGRES_HOUR_12_NAME = "hh";
+  public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12";
+  public static final String POSTGRES_HOUR_24_NAME = "hh24";
+  public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String POSTGRES_WEEK_OF_YEAR = "ww";
+  public static final String POSTGRES_MONTH = "mm";
+  public static final String POSTGRES_HALFDAY_AM = "am";
+  public static final String POSTGRES_HALFDAY_PM = "pm";
+
+  // jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, 
JODA_FULL_NAME_OF_DAY);
+postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+postgresToJod

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99317199
  
--- Diff: 
logical/src/test/java/org/apache/drill/common/expression/fn/JodaDateValidatorTest.java
 ---
@@ -0,0 +1,176 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static 
org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat;
+import static org.joda.time.DateTime.parse;
+import static org.joda.time.format.DateTimeFormat.forPattern;
+
+public class JodaDateValidatorTest {
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99312234
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/testing/TestDateConversions.java
 ---
@@ -0,0 +1,200 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.exec.fn.impl.testing;
+
+import org.apache.drill.BaseTestQuery;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+
+public class TestDateConversions extends BaseTestQuery {
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99312091
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java
 ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.drill.exec.expr.annotations.Workspace;
+
+<@pp.dropOutputFile/>
+
+<#list dateIntervalFunc.dates as type>
+
+<@pp.changeOutputFile name = 
"/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/>
+
+<#include "/@includes/license.ftl"/>
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.*;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+
+@FunctionTemplate(name = "sql_to_${type?lower_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class SqlTo${type} implements DrillSimpleFunc {
+
+@Param  VarCharHolder left;
+@Param  VarCharHolder right;
+@Workspace org.joda.time.format.DateTimeFormatter format;
+@Output ${type}Holder out;
+
+public void setup() {
+// Get the desired output format
+byte[] buf = new byte[right.end - right.start];
+right.buffer.getBytes(right.start, buf, 0, right.end - 
right.start);
+String formatString = new String(buf, 
com.google.common.base.Charsets.UTF_8);
+String pattern = 
org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat(formatString);
+format = org.joda.time.format.DateTimeFormat.forPattern(pattern);
+}
+
+public void eval() {
+// Get the input
+byte[] buf1 = new byte[left.end - left.start];
+left.buffer.getBytes(left.start, buf1, 0, left.end - left.start);
+String input = new String(buf1, 
com.google.common.base.Charsets.UTF_8);
+
+<#if type == "Date">
+out.value = (org.joda.time.DateMidnight.parse(input, 
format).withZoneRetainFields(org.joda.time.DateTimeZone.UTC)).getMillis();
+<#elseif type == "TimeStamp">
+out.value = org.joda.time.DateTime.parse(input, 
format).withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis();
+<#elseif type == "Time">
+out.value = (int) 
((format.parseDateTime(input)).withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis());
+
--- End diff --

Yes, we should. Fixed, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99311783
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java
 ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.drill.exec.expr.annotations.Workspace;
+
+<@pp.dropOutputFile/>
+
+<#list dateIntervalFunc.dates as type>
+
+<@pp.changeOutputFile name = 
"/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/>
+
+<#include "/@includes/license.ftl"/>
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.*;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+
+@FunctionTemplate(name = "sql_to_${type?lower_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class SqlTo${type} implements DrillSimpleFunc {
+
+@Param  VarCharHolder left;
+@Param  VarCharHolder right;
+@Workspace org.joda.time.format.DateTimeFormatter format;
+@Output ${type}Holder out;
+
+public void setup() {
+// Get the desired output format
+byte[] buf = new byte[right.end - right.start];
+right.buffer.getBytes(right.start, buf, 0, right.end - 
right.start);
+String formatString = new String(buf, 
com.google.common.base.Charsets.UTF_8);
+String pattern = 
org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat(formatString);
--- End diff --

Before the using Java 8 classes, we need change some patterns in 
JodaDateValidator.class, because there are some differences, but in general 
there shouldn't be user-visible change in behavior or semantics.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r99311060
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java
 ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.drill.exec.expr.annotations.Workspace;
+
+<@pp.dropOutputFile/>
+
+<#list dateIntervalFunc.dates as type>
+
+<@pp.changeOutputFile name = 
"/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/>
+
+<#include "/@includes/license.ftl"/>
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.*;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+
+@FunctionTemplate(name = "sql_to_${type?lower_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class SqlTo${type} implements DrillSimpleFunc {
+
+@Param  VarCharHolder left;
+@Param  VarCharHolder right;
+@Workspace org.joda.time.format.DateTimeFormatter format;
+@Output ${type}Holder out;
+
+public void setup() {
+// Get the desired output format
+byte[] buf = new byte[right.end - right.start];
+right.buffer.getBytes(right.start, buf, 0, right.end - 
right.start);
+String formatString = new String(buf, 
com.google.common.base.Charsets.UTF_8);
--- End diff --

I found the method that does the same. Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #735: DRILL-5237: Same query produces different/wrong res...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/735

DRILL-5237: Same query produces different/wrong results

Fixed loss of nested fields from the schema when the first 
FlattenRecordBatch is empty

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5237

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/735.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #735


commit 800953b7ec924ea31ffc21327950c84528f3c492
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2017-01-27T15:36:10Z

DRILL-5237: Same query produces different/wrong results




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5237) Same query produces different/wrong results

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5237:


 Summary: Same query produces different/wrong results
 Key: DRILL-5237
 URL: https://issues.apache.org/jira/browse/DRILL-5237
 Project: Apache Drill
  Issue Type: Bug
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Query 
{code:sql}
select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as b 
from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 'a1');
{code}
returns different results for different values for option 
planner.width.max_per_node
With options 
{code:sql}
alter session set `planner.slice_target` = 1;
alter session set `planner.width.max_per_node` = 1;
{code}
query returns correct result:
{noformat}
+--+
| col  |
+--+
| 3|
+--+
{noformat}
but with options 
{code:sql}
alter session set `planner.slice_target` = 1;
alter session set `planner.width.max_per_node` = 3;
{code}
the same query returns wrong result 
{noformat}
+--+
| col  |
+--+
| 2|
+--+
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill pull request #580: DRILL-4824: JSON with complex nested data produces ...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/580#discussion_r98232198
  
--- Diff: 
exec/vector/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java 
---
@@ -317,6 +317,12 @@ public Object getObject(int index) {
 if (v != null && index < v.getAccessor().getValueCount()) {
   Object value = v.getAccessor().getObject(index);
   if (value != null) {
+if ((v.getAccessor().getObject(index) instanceof Map
+&& ((Map) v.getAccessor().getObject(index)).size() == 
0)
+|| (v.getAccessor().getObject(index) instanceof List
+&& ((List) v.getAccessor().getObject(index)).size() == 
0)) {
+  continue;
+}
--- End diff --

@paul-rogers, map fields have data mode required and they are the part of 
the schema, that's why there are no difference between missing field in some 
record, and the field that exists but empty. 
This fix for your example will return result
`{"a":{"b":10},"c":[1,2,3]}`
`{"c":[4]}`
`{"b":[5]}`
`{"a":{"b":20}}`
`{"a":{"b":20}}`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #673: DRILL-4764: Parquet file with INT_16, etc. logical types n...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/673
  
Squashed all changes into single commit and rebased on master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #713: DRILL-3562: Query fails when using flatten on JSON data wh...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/713
  
@amansinha100, could you please review new changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #713: DRILL-3562: Query fails when using flatten on JSON ...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/713#discussion_r97576603
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -59,6 +59,12 @@
   private final boolean readNumbersAsDouble;
 
   /**
+   * Collection for tracking empty array writers during reading
+   * and storing them for initializing empty arrays
+   */
+  private final Set emptyArrayWritersSet = Sets.newHashSet();
--- End diff --

@amansinha100 Yes, you are right, in this place should be used List. Fixed 
it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #713: DRILL-3562: Query fails when using flatten on JSON ...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/713

DRILL-3562: Query fails when using flatten on JSON data where some do…

…cuments have an empty array
1. Added set for ListWriters tracking to keep empty arrays for further 
initializing in ensureAtLeastOneField method. 
2. Added check to avoid schema generating with field type "Late" and mode 
"Optional", replaced it to "Int" type in FlattenRecordBatch class.
3. Added unit tests to cover cases querying Json with empty arrays with 
flatten.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-3562

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #713


commit 815de0cc6f16b247b3a655007241a074e38394c7
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-12-20T16:55:41Z

DRILL-3562: Query fails when using flatten on JSON data where some 
documents have an empty array




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #711: DRILL-5164: Equi-join query results in CompileExcep...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/711

DRILL-5164: Equi-join query results in CompileException

1. Added method call rotateBlock() to avoid adding expressions into single 
Jblock.
2. Added unit tests to cover three types of joins: hash join, merge join, 
nested loop join.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5164

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/711.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #711


commit c448c5d4b4d8040dcf848b2478a3e833d9839d65
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-12-27T16:20:37Z

DRILL-5164: Equi-join query results in CompileException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-1808) Large compilation unit tests fails due to high memory allocation

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk resolved DRILL-1808.
--
   Resolution: Duplicate
Fix Version/s: (was: Future)
   1.10.0

Fixed in 
[810198b|https://github.com/apache/drill/commit/810198b18bfbe38712256c58b102bca079d934c1]

> Large compilation unit tests fails due to high memory allocation
> 
>
> Key: DRILL-1808
> URL: https://issues.apache.org/jira/browse/DRILL-1808
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Reporter: Steven Phillips
>Priority: Minor
> Fix For: 1.10.0
>
>
> The buildSchema method of external sort calls allocateNew() on each vector in 
> schema batch, which normally isn't much memory, but when there are thousands 
> of columns, this ends up being significant. We should avoid allocating any 
> memory for the schema batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
@chunhui-shi, I have made some changes, could you take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #654: DRILL-5032: Drill query on hive parquet table failed with ...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/654
  
@jinfengni, I have made some changes. Could you please review it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92801855
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map<String,String> parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map<String, String> partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map<String, 
String> parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
colum

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92801426
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map<String,String> parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map<String, String> partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map<String, 
String> parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
colum

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92800877
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map<String,String> parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map<String, String> partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map<String, 
String> parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
colum

[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/581
  
@paul-rogers, I have implemented changes that you have proposed. Could you 
take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r92478836
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -510,10 +517,13 @@ private void writeDataAllText(MapWriter map, 
FieldSelection selection,
   case VALUE_NUMBER_FLOAT:
   case VALUE_NUMBER_INT:
   case VALUE_STRING:
+removeNotNullColumn(fieldName);
--- End diff --

To clean 'path' if it knows it is recovering from some errors, was added 
this line: 
https://github.com/Serhii-Harnyk/drill/blob/9ce6d56b46bcd540697c85cd2f280831dd50b277/exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java#L243.
 Call of the method removeNotNullColumn() is a try to optimize the final size 
of the map fieldPathWriter, by removing fields, which were added to the map and 
initializing at current iteration.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #686: DRILL-5117: Compile error when query a json file with 1000...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/686
  
@sudheeshkatkam, when running for example test testEXTERNAL_SORT(), 
generates class CopierGen4, in which methods doSetup() and doEval() does not 
splits correctly. But with this fix them both splits into the smaller methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #686: DRILL-5117: Compile error when query a json file wi...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/686#discussion_r92452238
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/compile/TestLargeFileCompilation.java
 ---
@@ -154,4 +158,20 @@ public void testProject() throws Exception {
 testNoResult(ITERATION_COUNT, LARGE_QUERY_SELECT_LIST);
   }
 
+  @Test
+  public void testSelectAllFromFileWithManyColumns() throws Exception {
+File path = new File(BaseTestQuery.getTempDir("json/input"));
--- End diff --

You are right, with this changes both tests testEXTERNAL_SORT and 
testTOP_N_SORT work. So I enabled them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #686: DRILL-5117: Compile error when query a json file with 1000...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/686
  
@jinfengni, could you please review new changes? I squashed all changes 
into single commit and rebased into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #689: DRILL-4938: Need better error message

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/689
  
@zfong, I have added the unit test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #673: DRILL-4764: Parquet file with INT_16, etc. logical ...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/673#discussion_r91753304
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetFixedWidthDictionaryReaders.java
 ---
@@ -56,6 +58,31 @@ protected void readField(long recordsToReadInThisPass) {
 }
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
@chunhui-shi, could you please review new changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #686: DRILL-5117: Compile error when query a json file wi...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/686

DRILL-5117: Compile error when query a json file with 1000+columns



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5117

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/686.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #686


commit 00eaf30fd662530d8bd62059b85b0ad179768fdb
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-12-08T20:08:34Z

DRILL-5117: Compile error when query a json file with 1000+columns




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5117) Compile error when query a json file with 1000+columns

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5117:


 Summary: Compile error when query a json file with 1000+columns
 Key: DRILL-5117
 URL: https://issues.apache.org/jira/browse/DRILL-5117
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Codegen
Affects Versions: 1.8.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: Future


Query failed with compile error when we querying a json file with 1000+columns:
{noformat}
0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class 
"org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] 
(state=,code=0)
{noformat}

Stack trace from sqlline.log:
{noformat}
2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
27b54af4-b41f-0682-e50d-626de4eff68e: select * from 
dfs.`/tmp/tooManyFields.json` limit 1
2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
localhost.  Skipping affinity to that host.
2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs, Average 
start: 6.956000 μs .
2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State to report: RUNNING
2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN  
o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available - 
probably you're running Drill with a JRE and not a JDK
2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested RUNNING --> FAILED
2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested FAILED --> FINISHED
2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException: 
Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" grows 
beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
JaninoRuntimeException: Code attribute in class 
"org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.8.0.jar:1.8.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
 [drill-java-exec-1.8.0.jar:1.8.0]
at 
org.apache.drill.e

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r91351247
  
--- Diff: 
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/store/hive/schema/TestColumnListCache.java
 ---
@@ -0,0 +1,78 @@
+/*
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #673: DRILL-4764: Parquet file with INT_16, etc. logical types n...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/673
  
@parthchandra, could you please review new changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #654: DRILL-5032: Drill query on hive parquet table failed with ...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/654
  
@chunhui-shi, I have made some changes. Could you please review it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #673: DRILL-4764: Parquet file with INT_16, etc. logical types n...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/673
  
@parthchandra, I have made some changes and added few test values. Could 
you please review it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #673: DRILL-4764: Parquet file with INT_16, etc. logical ...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/673

DRILL-4764: Parquet file with INT_16, etc. logical types not supporte…

…d by simple SELECT

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-4764

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/673.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #673


commit ef8d7a294e1549eb5fbb61da3aad069a478ffca6
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-11-24T13:24:03Z

DRILL-4764: Parquet file with INT_16, etc. logical types not supported by 
simple SELECT




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r89520154
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java
 ---
@@ -398,12 +398,18 @@ public static void addConfToJob(final JobConf job, 
final Properties properties)
* Wrapper around {@link MetaStoreUtils#getPartitionMetadata(Partition, 
Table)} which also adds parameters from table
* to properties returned by {@link 
MetaStoreUtils#getPartitionMetadata(Partition, Table)}.
*
-   * @param partition
-   * @param table
-   * @return
+   * @param partition {@link Partition} instance
+   * @param table {@link Table} instance
+   * @return properties
*/
   public static Properties getPartitionMetadata(final Partition partition, 
final Table table) {
-final Properties properties = 
MetaStoreUtils.getPartitionMetadata(partition, table);
+final Properties properties;
--- End diff --

@chunhui-shi let me clarify. Do you propose something like creating 
"partition groups" and storing columns in this "groups"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #657: DRILL-5048: AssertionError when case statement is u...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/657

DRILL-5048: AssertionError when case statement is used with timestamp…

… and null

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5048

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/657.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #657


commit 66823553ec9074ee0adbb93f02cc86f41040d81b
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-11-17T14:23:58Z

DRILL-5048: AssertionError when case statement is used with timestamp and 
null




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5048) AssertionError when case statement is used with timestamp and null

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5048:


 Summary: AssertionError when case statement is used with timestamp 
and null
 Key: DRILL-5048
 URL: https://issues.apache.org/jira/browse/DRILL-5048
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: Future


AssertionError when we use case with timestamp and null:

{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN 
CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
. . . . . . . . . . . . . . > FROM
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE 
CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res
. . . . . . . . . . . . . . > FROM (values(1)) foo
. . . . . . . . . . . . . . > ) foobar;
Error: SYSTEM ERROR: AssertionError: Type mismatch:
rowtype of new rel:
RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
rowtype of set:
RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL


[Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log

{noformat}
Caused by: java.lang.AssertionError: Type mismatch:
rowtype of new rel:
RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
rowtype of set:
RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
... 16 common frames omitted
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/654

DRILL-5032: Drill query on hive parquet table failed with OutOfMemoryError



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5032

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/654.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #654


commit 90482d7a1be99293fc3afdf2a297ee08e8831f66
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-10-27T19:20:27Z

DRILL-5032 Drill query on hive parquet table failed with OutOfMemoryError: 
Java heap space




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5032:


 Summary: Drill query on hive parquet table failed with 
OutOfMemoryError: Java heap space
 Key: DRILL-5032
 URL: https://issues.apache.org/jira/browse/DRILL-5032
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Hive
Affects Versions: 1.8.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Following query on hive parquet table failed with OOM Java heap space:
{code}
select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
vmdr_trades where trade_date='2016-04-12'
2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 1 ms
2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 3 ms
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 0 ms
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 0 ms
2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 0 ms
2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 0 ms
2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. 
Information message: Unable to handle out of memory condition in Foreman.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) 
~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
 ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:166) ~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74]
at 
com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat.print(TextFormat.java:71) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat.printToString(TextFormat.java:118) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:106) 
~[protobuf-java-2.5.0.jar:na

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
@chunhui-shi 
Results of performance test

| % of nullable records | with fix | master | %(t-t0)/t0 |
| 0.001 | 13986,8 | 13752,1 | 1,71% |
| 0.1 | 13873,6 | 13081,3 | 6,06% |
| 0.5 | 11345,6 | 7552,44 | 50,22% |
| 0.7 | 12699,1 | 5753,44 | 120,72% |
| 0.99 | 14544,2 | 494 | 2844,17% |

So increase of performance depends on % of nullable fields in dataset


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
@chunhui-shi what about second commit?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: TO_TIMESTAMP function returns in-correct results

[Serhii Harnyk]

Example of the usage with "s" for second of minute and "S" for fraction of
second:

VALUES(TO_TIMESTAMP('2015-03-30 20:49:59.10 UTC', '-MM-dd HH:mm:ss.SSS
z'))

2016-10-28 16:17 GMT+03:00 Khurram Faraaz <kfar...@maprtech.com>:

> Thanks Serhii.
>
> Can you please give me a working example of the usage with "s" for second
> of minute and "S" for fraction of second.
>
> I tried with both those symbols, however Drill 1.9.0 (commit: a29f1e29)
> does not honor those symbols when used from within the to_date function.
>
> On Thu, Oct 27, 2016 at 6:31 PM, Serhii Harnyk <serhii.har...@gmail.com>
> wrote:
>
> > Hello, Khurram
> >
> > http://joda-time.sourceforge.net/apidocs/org/joda/time/
> > format/DateTimeFormat.html
> >
> > s   second of minute number55
> > S   fraction of second   number978
> >
> >
> >
> > 2016-10-27 13:54 GMT+03:00 Khurram Faraaz <kfar...@maprtech.com>:
> >
> > > All,
> > >
> > > I am on Drill 1.9.0 git commit ID : a29f1e29 on CentOS
> > >
> > > TO_TIMESTAMP function does not return correct results, note that the
> > > minutes, seconds and milliseconds parts of timestamp are incorrect in
> the
> > > results
> > >
> > > {noformat}
> > > 0: jdbc:drill:schema=dfs.tmp> VALUES(TO_TIMESTAMP('2015-03-30
> > 20:49:59.10
> > > UTC', '-MM-dd HH:mm:ss.s z'));
> > > ++
> > > | EXPR$0 |
> > > ++
> > > | 2015-03-30 20:49:10.0  |
> > > ++
> > > 1 row selected (0.228 seconds)
> > > {noformat}
> > >
> > > {noformat}
> > > 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST(TO_TIMESTAMP('2015-03-30
> > > 20:49:59.10 UTC', '-MM-dd HH:mm:ss.s z') AS TIMESTAMP));
> > > ++
> > > | EXPR$0 |
> > > ++
> > > | 2015-03-30 20:49:10.0  |
> > > ++
> > > 1 row selected (0.265 seconds)
> > > {noformat}
> > >
> > > This case returns correct results, when the same string used above is
> > given
> > > as input to CAST function, note that minutes mm, seconds ss and
> > millisecond
> > > s parts are honored
> > >
> > > {noformat}
> > > 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST('2015-03-30 20:49:59.10 UTC'
> AS
> > > TIMESTAMP));
> > > ++
> > > | EXPR$0 |
> > > ++
> > > | 2015-03-30 20:49:59.1  |
> > > ++
> > > 1 row selected (0.304 seconds)
> > > {noformat}
> > >
> > > Thanks,
> > > Khurram
> > >
> >
>

Re: TO_TIMESTAMP function returns in-correct results

[Serhii Harnyk]

Hello, Khurram

http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html

s   second of minute number55
S   fraction of second   number978



2016-10-27 13:54 GMT+03:00 Khurram Faraaz :

> All,
>
> I am on Drill 1.9.0 git commit ID : a29f1e29 on CentOS
>
> TO_TIMESTAMP function does not return correct results, note that the
> minutes, seconds and milliseconds parts of timestamp are incorrect in the
> results
>
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> VALUES(TO_TIMESTAMP('2015-03-30 20:49:59.10
> UTC', '-MM-dd HH:mm:ss.s z'));
> ++
> | EXPR$0 |
> ++
> | 2015-03-30 20:49:10.0  |
> ++
> 1 row selected (0.228 seconds)
> {noformat}
>
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST(TO_TIMESTAMP('2015-03-30
> 20:49:59.10 UTC', '-MM-dd HH:mm:ss.s z') AS TIMESTAMP));
> ++
> | EXPR$0 |
> ++
> | 2015-03-30 20:49:10.0  |
> ++
> 1 row selected (0.265 seconds)
> {noformat}
>
> This case returns correct results, when the same string used above is given
> as input to CAST function, note that minutes mm, seconds ss and millisecond
> s parts are honored
>
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST('2015-03-30 20:49:59.10 UTC' AS
> TIMESTAMP));
> ++
> | EXPR$0 |
> ++
> | 2015-03-30 20:49:59.1  |
> ++
> 1 row selected (0.304 seconds)
> {noformat}
>
> Thanks,
> Khurram
>

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r84509937
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -337,6 +364,7 @@ private void writeData(MapWriter map, FieldSelection 
selection, boolean moveForw
   continue outside;
 }
 
+nullableFields.remove(fieldName);
--- End diff --

Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
Added fixes according to review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r84509944
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -418,6 +447,7 @@ private void writeDataAllText(MapWriter map, 
FieldSelection selection, boolean m
 continue outside;
   }
 
+  nullableFields.remove(fieldName);
--- End diff --

Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r84507055
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -56,6 +58,12 @@
   private final boolean readNumbersAsDouble;
 
   /**
+   * Collection for tracking nullable fields during reading
+   * and storing them for creating default typed vectors
+   */
+  private final Set nullableFields  = Sets.newLinkedHashSet();
--- End diff --

We need to store an order of nullable fields.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r84507516
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -131,14 +139,33 @@ public void ensureAtLeastOneField(ComplexWriter 
writer) {
   PathSegment fieldPath = fieldPathList.get(j);
   if (emptyStatus.get(j)) {
 if (allTextMode) {
-  fieldWriter.varChar(fieldPath.getNameSegment().getPath());
+  if (checkNullFields(fieldPathList)) {
--- End diff --

Added fixes for nested fields with handling "path" of the field. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/594#discussion_r84508112
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
 ---
@@ -131,14 +139,33 @@ public void ensureAtLeastOneField(ComplexWriter 
writer) {
   PathSegment fieldPath = fieldPathList.get(j);
   if (emptyStatus.get(j)) {
 if (allTextMode) {
-  fieldWriter.varChar(fieldPath.getNameSegment().getPath());
+  if (checkNullFields(fieldPathList)) {
+for (String fieldName : nullableFields) {
+  fieldWriter.varChar(fieldName);
+}
+  } else {
+fieldWriter.varChar(fieldPath.getNameSegment().getPath());
+  }
 } else {
-  fieldWriter.integer(fieldPath.getNameSegment().getPath());
+  if (checkNullFields(fieldPathList)) {
--- End diff --

Test added


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/581
  
@adeneche - miliseconds issue is fixed in last commit


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83871818
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,213 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set ansiValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap ansiToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  //tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  //ansi patterns
+  public static final String ANSI_FULL_NAME_OF_DAY = "day";
+  public static final String ANSI_DAY_OF_YEAR = "ddd";
+  public static final String ANSI_DAY_OF_MONTH = "dd";
+  public static final String ANSI_DAY_OF_WEEK = "d";
+  public static final String ANSI_NAME_OF_MONTH = "month";
+  public static final String ANSI_ABR_NAME_OF_MONTH = "mon";
+  public static final String ANSI_FULL_ERA_NAME = "ee";
+  public static final String ANSI_NAME_OF_DAY = "dy";
+  public static final String ANSI_TIME_ZONE_NAME = "tz";
+  public static final String ANSI_HOUR_12_NAME = "hh";
+  public static final String ANSI_HOUR_12_OTHER_NAME = "hh12";
+  public static final String ANSI_HOUR_24_NAME = "hh24";
+  public static final String ANSI_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String ANSI_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String ANSI_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String ANSI_WEEK_OF_YEAR = "ww";
+  public static final String ANSI_MONTH = "mm";
+  public static final String ANSI_HALFDAY_AM = "am";
+  public static final String ANSI_HALFDAY_PM = "pm";
+
+  //jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "S";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+ansiToJodaMap.put(ANSI_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY);
+ansiToJodaMap.put(ANSI_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+ansiToJodaMap.put(ANSI_DAY_OF_MONTH, JODA_DAY_OF_MONTH);
+ansiToJodaMap.put(ANSI_DAY_OF_WEEK

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83871606
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
 ---
@@ -408,6 +411,12 @@ private LogicalExpression 
getDrillFunctionFromOptiqCall(RexCall call) {
 
   return first;
 }
+  } else if (functionName.equals("to_date") || 
functionName.equals("to_time") || functionName.equals("to_timestamp")) {
+// convert ansi date format string to joda according to session 
option
+OptionManager om = this.context.getPlannerSettings().getOptions();
+
if(ToDateFormats.valueOf(om.getOption(ExecConstants.TO_DATE_FORMAT).string_val.toUpperCase()).equals(ToDateFormats.ANSI))
 {
--- End diff --

Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83871870
  
--- Diff: 
logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java
 ---
@@ -0,0 +1,213 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.common.expression.fn;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+
+import java.util.Comparator;
+import java.util.Set;
+
+public class JodaDateValidator {
+
+  private static final Set ansiValuesForDeleting = 
Sets.newTreeSet(new LengthDescComparator());
+  private static final CaseInsensitiveMap ansiToJodaMap = 
CaseInsensitiveMap.newTreeMap(new LengthDescComparator());
+
+  //tokens for deleting
+  public static final String SUFFIX_SP = "sp";
+  public static final String PREFIX_FM = "fm";
+  public static final String PREFIX_FX = "fx";
+  public static final String PREFIX_TM = "tm";
+
+  //ansi patterns
+  public static final String ANSI_FULL_NAME_OF_DAY = "day";
+  public static final String ANSI_DAY_OF_YEAR = "ddd";
+  public static final String ANSI_DAY_OF_MONTH = "dd";
+  public static final String ANSI_DAY_OF_WEEK = "d";
+  public static final String ANSI_NAME_OF_MONTH = "month";
+  public static final String ANSI_ABR_NAME_OF_MONTH = "mon";
+  public static final String ANSI_FULL_ERA_NAME = "ee";
+  public static final String ANSI_NAME_OF_DAY = "dy";
+  public static final String ANSI_TIME_ZONE_NAME = "tz";
+  public static final String ANSI_HOUR_12_NAME = "hh";
+  public static final String ANSI_HOUR_12_OTHER_NAME = "hh12";
+  public static final String ANSI_HOUR_24_NAME = "hh24";
+  public static final String ANSI_MINUTE_OF_HOUR_NAME = "mi";
+  public static final String ANSI_SECOND_OF_MINUTE_NAME = "ss";
+  public static final String ANSI_MILLISECOND_OF_MINUTE_NAME = "ms";
+  public static final String ANSI_WEEK_OF_YEAR = "ww";
+  public static final String ANSI_MONTH = "mm";
+  public static final String ANSI_HALFDAY_AM = "am";
+  public static final String ANSI_HALFDAY_PM = "pm";
+
+  //jodaTime patterns
+  public static final String JODA_FULL_NAME_OF_DAY = "";
+  public static final String JODA_DAY_OF_YEAR = "D";
+  public static final String JODA_DAY_OF_MONTH = "d";
+  public static final String JODA_DAY_OF_WEEK = "e";
+  public static final String JODA_NAME_OF_MONTH = "";
+  public static final String JODA_ABR_NAME_OF_MONTH = "MMM";
+  public static final String JODA_FULL_ERA_NAME = "G";
+  public static final String JODA_NAME_OF_DAY = "E";
+  public static final String JODA_TIME_ZONE_NAME = "TZ";
+  public static final String JODA_HOUR_12_NAME = "h";
+  public static final String JODA_HOUR_12_OTHER_NAME = "h";
+  public static final String JODA_HOUR_24_NAME = "H";
+  public static final String JODA_MINUTE_OF_HOUR_NAME = "m";
+  public static final String JODA_SECOND_OF_MINUTE_NAME = "s";
+  public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "S";
+  public static final String JODA_WEEK_OF_YEAR = "w";
+  public static final String JODA_MONTH = "MM";
+  public static final String JODA_HALFDAY = "aa";
+
+  static {
+ansiToJodaMap.put(ANSI_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY);
+ansiToJodaMap.put(ANSI_DAY_OF_YEAR, JODA_DAY_OF_YEAR);
+ansiToJodaMap.put(ANSI_DAY_OF_MONTH, JODA_DAY_OF_MONTH);
+ansiToJodaMap.put(ANSI_DAY_OF_WEEK

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83871748
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
 ---
@@ -408,6 +411,12 @@ private LogicalExpression 
getDrillFunctionFromOptiqCall(RexCall call) {
 
   return first;
 }
+  } else if (functionName.equals("to_date") || 
functionName.equals("to_time") || functionName.equals("to_timestamp")) {
+// convert ansi date format string to joda according to session 
option
+OptionManager om = this.context.getPlannerSettings().getOptions();
+
if(ToDateFormats.valueOf(om.getOption(ExecConstants.TO_DATE_FORMAT).string_val.toUpperCase()).equals(ToDateFormats.ANSI))
 {
+  args.set(1, FunctionCallFactory.createExpression("ansi_to_joda", 
Arrays.asList(args.get(1;
--- End diff --

Yes, it would be two nested ansi_to_joda conversions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83871584
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
 ---
@@ -408,6 +411,12 @@ private LogicalExpression 
getDrillFunctionFromOptiqCall(RexCall call) {
 
   return first;
 }
+  } else if (functionName.equals("to_date") || 
functionName.equals("to_time") || functionName.equals("to_timestamp")) {
--- End diff --

No. Variable functionName has been already lowercased. It's for all 
functions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83870798
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/AnsiToJoda.java 
---
@@ -0,0 +1,58 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+/**
+ * Replaces all ansi patterns to joda equivalents.
+ */
+@FunctionTemplate(name = "ansi_to_joda",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls= FunctionTemplate.NullHandling.NULL_IF_NULL)
--- End diff --

Fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/581#discussion_r83871296
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/AnsiToJoda.java 
---
@@ -0,0 +1,58 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+/**
+ * Replaces all ansi patterns to joda equivalents.
+ */
+@FunctionTemplate(name = "ansi_to_joda",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls= FunctionTemplate.NullHandling.NULL_IF_NULL)
+public class AnsiToJoda implements DrillSimpleFunc {
+
+  @Param
+  VarCharHolder in;
+
+  @Output
+  VarCharHolder out;
+
+  @Inject
+  DrillBuf buffer;
+
+  @Override
+  public void setup() {
+  }
+
+  @Override
+  public void eval() {
+String pattern = 
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(in.start,
 in.end, in.buffer);
--- End diff --

Datetime patterns validation isn't a simple task. For example JODA Time has 
no validation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #598: DRILL-4906 CASE Expression with constant generates ...

[Serhii-Harnyk]

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/598#discussion_r81115837
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/planner/logical/TestCaseNullableTypes.java
 ---
@@ -0,0 +1,107 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to you under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.drill.BaseTestQuery;
+import org.junit.Test;
+
+/**
+ * DRILL-4906
+ * Tests for handling nullable types in CASE function
+ */
+public class TestCaseNullableTypes extends BaseTestQuery {
+
+  @Test
+  public void testCaseNullableTypesInt() throws Exception {
+testBuilder()
+.sqlQuery("select (case when (false) then null else 1 end) res1 
from (values(1))")
+.unOrdered()
+.baselineColumns("res1")
+.baselineValues(1)
+.go();
+  }
+
+  @Test
+  public void testCaseNullableTypesVarchar() throws Exception {
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #598: DRILL-4906 CASE Expression with constant generates ...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/598

DRILL-4906 CASE Expression with constant generates class exception



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-4906

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/598.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #598


commit 46f0df7cf89fff8e7d8cf9a21a810e1b4e292bce
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-09-22T12:06:10Z

DRILL-4906 CASE Expression with constant generates class exception




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4906) CASE Expression with constant generates class exception

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-4906:


 Summary: CASE Expression with constant generates class exception
 Key: DRILL-4906
 URL: https://issues.apache.org/jira/browse/DRILL-4906
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser
Affects Versions: 1.8.0, 1.6.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: 1.9.0


How to reproduce:

select (case when (true) then 1 end) from (values(1));

Error
Error: SYSTEM ERROR: ClassCastException: 
org.apache.drill.exec.expr.holders.NullableVarCharHolder cannot be cast to 
org.apache.drill.exec.expr.holders.VarCharHolder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/594

DRILL-4842: SELECT * on JSON data results in NumberFormatException



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-4842

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/594.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #594


commit 190a69a65ad8c144b164c2acacf9718a0ecb3768
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-09-08T18:11:37Z

DRILL-4842: SELECT * on JSON data results in NumberFormatException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/581

DRILL-4864: Add ANSI format for date/time functions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-4864

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/581.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #581


commit e580e38290de4e3a4f7510811e3c5a956c5d7594
Author: Serhii-Harnyk <serhii.har...@gmail.com>
Date:   2016-09-01T14:48:02Z

DRILL-4864: Add ANSI format for date/time functions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---