[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[paul-rogers]

Github user paul-rogers closed the pull request at:

https://github.com/apache/drill/pull/866


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #866: DRILL-5657: Implement size-aware result set loader

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/866
  
Closing as this PR is now superseded by #914.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #887: DRILL-5688: Add repeated map support to column acce...

[paul-rogers]

Github user paul-rogers closed the pull request at:

https://github.com/apache/drill/pull/887


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #887: DRILL-5688: Add repeated map support to column accessors

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/887
  
Closing as this PR is now superseded by #914.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #914: DRILL-5657: Size-aware vector writer structure

[paul-rogers]

GitHub user paul-rogers opened a pull request:

https://github.com/apache/drill/pull/914

DRILL-5657: Size-aware vector writer structure

This large PR provides another two levels of foundation for size-aware 
vector writers in the Drill record readers. It combines code from two previous 
PRS:

* PR 866 - DRILL-5657: Implement size-aware result set loader
* PR 887 - DRILL-5688: Add repeated map support to column accessors

The PR then goes on to integrate the two prior PRs and provide additional 
functionality.

Like the two previous PRs, this one is divided into commits to group the 
work.

1. Accessor layer
2. Row Set layer
3. Tuple and Column Model layer
4. Row Set Loader layer
5. Secondary changes

Much of the material below appears in Javadoc throughout the code. The 
material here is not meant to replace that documentation. Instead, it is meant 
to provide the “big picture”: placing the bits and pieces in context and 
pointing out interesting functionality to explore in each layer.

## Commit 1: The Accessor Layer

The first commit provides the core of the mechanism: the writers that put 
data into vectors, and the readers that retrieve that data. The version here is 
an evolution of the version provided in an earlier PR a few months ago.

### Overview of the Drill Vector Data Model

The code will make much more sense if we start with a review of Drill’s 
complex vector data model. Drill has 38+ data (“minor”) types as defined in 
the [proto 
buf](https://github.com/apache/drill/blob/master/protocol/src/main/protobuf/Types.proto)
 definition. Drill also has three cardinalities (“modes”) defined in the 
same file. The result is over 120+ different vector types. Then, when you add 
maps, repeated maps, lists and repeated lists, you rapidly get an explosion of 
types that the writer code must handle.

Vectors can be categorized along multiple dimensions:

* By data (minor) type
* By cardinality (mode)
* By fixed or variable width

A repeated map, a list, a repeated list and any array (repeated) scalar all 
are array-like. Nullable and required modes are identical (single values), but 
a nullable has an additional is-set (“bit”) vector.

A key contribution of this PR is the data model used to organize vectors.

* Both the top-level row, and a Drill map are “tuples” and are treated 
similarly in the model.
* All non-map, non-list (that is, scalar) data types are treated uniformly.
* All arrays (whether a list, a repeated list, a repeated map, or a 
repeated scalar) are treated uniformly.

### Accessor Data Model

The above leads to a very simple, JSON-like data model, introduced in this 
PR.

* A tuple reader or writer models a row. (Usually via a subclass.) Column 
are accessible by name or position.
* Every column is modeled as an object.
* The object can have an object type: scalar, tuple or array.
* An array has a single element type (but many run-time elements)
* A scalar can be nullable or not, and provides a uniform get/set interface.

This data model is similar to; but has important differences from, the 
prior, generated, readers and writers. 

The object layer is new: it is the simplest way to model the three 
“object types.” An app using this code would use just the leaf scalar 
readers and writers.

Although there is quite a bit of code change here to provide the new 
structure the core functionality of reading and writing to vectors has not 
changed much. And, this code has extensive unit tests, which should avoid the 
need to "mentally execute" each line of code.

See the classes in `org.apache.drill.exec.vector.accessor` for details. In 
particular, please see the `package-info.java` file in that package for more 
information.

As before, the top package provides a set of interfaces; the inner packages 
provide the implementation. The `ColumnAccessors.java` template generates the 
per-vector code. Warning: this template has become quite cryptic: the best bet 
for review is to generate the Java code and review that.

### Writer Performance

During previous review, we discussed ways to optimize writer performance. 
This PR has two improvements:

* Completely rework the writers to minimize code steps
* Rework the “column loaders” to eliminate them: instead of two 
additional method calls, the “loader” now uses the column writers directly.

Much behind-the-scenes rework was needed to accomplish the above.

Column readers, however, were left with their existing structure; we can 
apply the same optimizations to the readers in a later PR.

While doing the performance optimization, it became clear we can adopt a 
major simplification. Writers evolved from having three versions 

[GitHub] drill pull request #907: DRILL-5697: Improve performance of filter operator ...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/907#discussion_r133859807
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/RegexpUtil.java 
---
@@ -96,20 +145,46 @@ public static String sqlToRegexLike(
 || (nextChar == '%')
 || (nextChar == escapeChar)) {
   javaPattern.append(nextChar);
+  simplePattern.append(nextChar);
   i++;
 } else {
   throw invalidEscapeSequence(sqlPattern, i);
 }
   } else if (c == '_') {
+// if we find _, it is not simple pattern, we are looking for only 
%
+notSimple = true;
 javaPattern.append('.');
   } else if (c == '%') {
+if (i == 0) {
+  // % at the start could potentially be one of the simple cases 
i.e. ENDS_WITH.
+  endsWith = true;
+} else if (i == (len-1)) {
+  // % at the end could potentially be one of the simple cases 
i.e. STARTS_WITH
+  startsWith = true;
+} else {
+  // If we find % anywhere other than start or end, it is not a 
simple case.
--- End diff --

Consider ABC%XYZ.
It might be worthwhile to decide whether to leverage a pattern or fall back 
to Java's Regex util based on the number of occurrences of '%' as a criteria.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #907: DRILL-5697: Improve performance of filter operator ...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/907#discussion_r133859377
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/RegexpUtil.java 
---
@@ -47,18 +47,55 @@
   "[:alnum:]", "\\p{Alnum}"
   };
 
+  // type of pattern string.
+  public enum sqlPatternType {
+STARTS_WITH, // Starts with a constant string followed by any string 
values (ABC%)
+ENDS_WITH, // Ends with a constant string, starts with any string 
values (%ABC)
+CONTAINS, // Contains a constant string, starts and ends with any 
string values (%ABC%)
--- End diff --

You should add a pattern of the form 'Starts with a constant, ends with 
another constant, and has any string in between'
(ABC%XYZ)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-5253) External sort fails with OOM error (Fails to allocate sv2)

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers resolved DRILL-5253.

   Resolution: Fixed
Fix Version/s: 1.12.0

> External sort fails with OOM error (Fails to allocate sv2)
> --
>
> Key: DRILL-5253
> URL: https://issues.apache.org/jira/browse/DRILL-5253
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0
>Reporter: Rahul Challapalli
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Attachments: 2762f36d-a2e7-5582-922d-3c4626be18c0.sys.drill
>
>
> git.commit.id.abbrev=2af709f
> The data set used in the below query has the same value for every column in 
> every row. The query fails with an OOM as it exceeds the allocated memory
> {code}
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 104857600;
>  select count(*) from (select * from identical order by col1, col2, col3, 
> col4, col5, col6, col7, col8, col9, col10);
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate sv2 
> buffer after repeated attempts
> Fragment 2:0
> [Error Id: aed43fa1-fd8b-4440-9426-0f35d055aabb on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Exception from the logs
> {code}
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
> nodes ran out of memory while executing the query.
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate sv2 
> buffer after repeated attempts
> [Error Id: aed43fa1-fd8b-4440-9426-0f35d055aabb ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
>  ~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:242)
>  [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_111]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: 
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate sv2 
> buffer after repeated attempts
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:371)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
> ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:92)
>  ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
> ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> 

[jira] [Resolved] (DRILL-5268) SYSTEM ERROR: UnsupportedOperationException: Unable to get size for minor type [MAP] and mode [REQUIRED]

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers resolved DRILL-5268.

   Resolution: Fixed
Fix Version/s: 1.12.0

> SYSTEM ERROR: UnsupportedOperationException: Unable to get size for minor 
> type [MAP] and mode [REQUIRED]
> 
>
> Key: DRILL-5268
> URL: https://issues.apache.org/jira/browse/DRILL-5268
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.10.0
>Reporter: Rahul Challapalli
> Fix For: 1.12.0
>
> Attachments: drill5268.tgz
>
>
> git.commit.id.abbrev=300e934
> With the managed external sort turned on, I get the below error
> {code}
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.memory.max_query_memory_per_node` = 52428800;
> select * from (select d1.type, d1.evnt, d1.transaction from (select d.type 
> type, flatten(d.events) evnt, flatten(d.transactions) transaction from 
> dfs.`/drill/testdata/resource-manager/10rows/data.json` d) d1 order by 
> d1.evnt.event_time, d1.transaction.trans_time) d2 where d2.type='web' and 
> d2.evnt.evnt_type = 'cmpgn4';
> Error: SYSTEM ERROR: UnsupportedOperationException: Unable to get size for 
> minor type [MAP] and mode [REQUIRED]
> Fragment 0:0
> [Error Id: a9dc1de5-2ff7-44db-bdd2-b166b1f0cea8 on qa-node183.qa.lab:31010] 
> (state=,code=0)
> {code}
> If we do not enable the managed sort, then we end up with DRILL-5234



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Failing Tests In Master

[Timothy Farkas]

They're failing on my mac with JDK 1.7.

Pritesh asked me to look into the JDK 8 failures. But it seems like something 
additional is going wrong in my case. I'll investigate this as part of  
DRILL-5730

Thanks,
Tim


From: Padma Penumarthy 
Sent: Thursday, August 17, 2017 4:51:02 PM
To: dev@drill.apache.org
Subject: Re: Failing Tests In Master

These tests fail on my mac with JDK 1.8.

> On Aug 17, 2017, at 4:42 PM, Jinfeng Ni  wrote:
>
> Running on CentOS (JDK1.7) with latest master against those unit testcase,
> and I did not see errors.  Any difference in terms of OS/JDK?
>
> git log
> commit e9065b55ea560e7f737d6fcb4948f9e945b9b14f
>
> mvn -DfailIfNoTests=false
> -Dtest=TestMetadataProvider,TestCustomUserAuthenticator,TestInfoSchema,TestViewSupport,TestParquetScan
> surefire:test
>
> Running org.apache.drill.exec.work.metadata.TestMetadataProvider
> Running
> org.apache.drill.exec.work.metadata.TestMetadataProvider#columnsWithColumnNameFilter
> Running org.apache.drill.exec.sql.TestViewSupport
> ...
>
> Results :
>
> Tests run: 82, Failures: 0, Errors: 0, Skipped: 1
>
>
> On Thu, Aug 17, 2017 at 3:59 PM, Padma Penumarthy 
> wrote:
>
>> Yes, I do see these failures.
>>
>> Thanks,
>> Padma
>>
>>> On Aug 17, 2017, at 3:55 PM, Timothy Farkas  wrote:
>>>
>>> These tests are consistently failing for me with the latest master. Does
>> anyone else see this issue?
>>>
>>>
>>> Failed tests:
>>>
>>> TestMetadataProvider.tables:153 expected: but was:
>>>
>>> TestMetadataProvider.tablesWithTableNameFilter:212 expected: but
>> was:
>>>
>>> TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but
>> was:
>>>
>>> TestMetadataProvider.tablesWithTableFilter:176 expected: but
>> was:
>>>
>>>
>>> Tests in error:
>>>
>>> TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM
>> ERROR: URISyn...
>>>
>>> TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth »
>> UserRemote
>>>
>>> TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR:
>> URISyntaxExcepti...
>>>
>>> TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344
>> » Rpc
>>>
>>> TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote
>> SYSTEM E...
>>>
>>> TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344
>> » Rpc o...
>>>
>>> Thanks,
>>> Tim
>>>
>>
>>

[jira] [Created] (DRILL-5730) Fix Unit Test failures on JDK 8 And Some JDK 7 versions

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-5730:
-

 Summary: Fix Unit Test failures on JDK 8 And Some JDK 7 versions
 Key: DRILL-5730
 URL: https://issues.apache.org/jira/browse/DRILL-5730
 Project: Apache Drill
  Issue Type: Bug
Reporter: Timothy Farkas
Assignee: Timothy Farkas


Tests fail on JDK 8 and oracle JDK 7 on my mac

Failed tests: 
  TestMetadataProvider.tables:153 expected: but was:
  TestMetadataProvider.tablesWithTableNameFilter:212 expected: but 
was:
  TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but 
was:
  TestMetadataProvider.tablesWithTableFilter:176 expected: but was:

Tests in error: 
  TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR: 
URISyntaxExcepti...
  TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM ERROR: 
URISyn...
  TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth » UserRemote
  TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344 » 
Rpc
  TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344 » Rpc 
o...






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill issue #913: - DRILL-5729 Fix Travis Build

[ilooner-mapr]

Github user ilooner-mapr commented on the issue:

https://github.com/apache/drill/pull/913
  
@parthchandra 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Failing Tests In Master

[Padma Penumarthy]

These tests fail on my mac with JDK 1.8.

> On Aug 17, 2017, at 4:42 PM, Jinfeng Ni  wrote:
> 
> Running on CentOS (JDK1.7) with latest master against those unit testcase,
> and I did not see errors.  Any difference in terms of OS/JDK?
> 
> git log
> commit e9065b55ea560e7f737d6fcb4948f9e945b9b14f
> 
> mvn -DfailIfNoTests=false
> -Dtest=TestMetadataProvider,TestCustomUserAuthenticator,TestInfoSchema,TestViewSupport,TestParquetScan
> surefire:test
> 
> Running org.apache.drill.exec.work.metadata.TestMetadataProvider
> Running
> org.apache.drill.exec.work.metadata.TestMetadataProvider#columnsWithColumnNameFilter
> Running org.apache.drill.exec.sql.TestViewSupport
> ...
> 
> Results :
> 
> Tests run: 82, Failures: 0, Errors: 0, Skipped: 1
> 
> 
> On Thu, Aug 17, 2017 at 3:59 PM, Padma Penumarthy 
> wrote:
> 
>> Yes, I do see these failures.
>> 
>> Thanks,
>> Padma
>> 
>>> On Aug 17, 2017, at 3:55 PM, Timothy Farkas  wrote:
>>> 
>>> These tests are consistently failing for me with the latest master. Does
>> anyone else see this issue?
>>> 
>>> 
>>> Failed tests:
>>> 
>>> TestMetadataProvider.tables:153 expected: but was:
>>> 
>>> TestMetadataProvider.tablesWithTableNameFilter:212 expected: but
>> was:
>>> 
>>> TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but
>> was:
>>> 
>>> TestMetadataProvider.tablesWithTableFilter:176 expected: but
>> was:
>>> 
>>> 
>>> Tests in error:
>>> 
>>> TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM
>> ERROR: URISyn...
>>> 
>>> TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth »
>> UserRemote
>>> 
>>> TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR:
>> URISyntaxExcepti...
>>> 
>>> TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344
>> » Rpc
>>> 
>>> TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote
>> SYSTEM E...
>>> 
>>> TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344
>> » Rpc o...
>>> 
>>> Thanks,
>>> Tim
>>> 
>> 
>> 

Re: Failing Tests In Master

[Jinfeng Ni]

Running on CentOS (JDK1.7) with latest master against those unit testcase,
and I did not see errors.  Any difference in terms of OS/JDK?

git log
commit e9065b55ea560e7f737d6fcb4948f9e945b9b14f

mvn -DfailIfNoTests=false
-Dtest=TestMetadataProvider,TestCustomUserAuthenticator,TestInfoSchema,TestViewSupport,TestParquetScan
surefire:test

Running org.apache.drill.exec.work.metadata.TestMetadataProvider
Running
org.apache.drill.exec.work.metadata.TestMetadataProvider#columnsWithColumnNameFilter
Running org.apache.drill.exec.sql.TestViewSupport
...

Results :

Tests run: 82, Failures: 0, Errors: 0, Skipped: 1


On Thu, Aug 17, 2017 at 3:59 PM, Padma Penumarthy 
wrote:

> Yes, I do see these failures.
>
> Thanks,
> Padma
>
> > On Aug 17, 2017, at 3:55 PM, Timothy Farkas  wrote:
> >
> > These tests are consistently failing for me with the latest master. Does
> anyone else see this issue?
> >
> >
> > Failed tests:
> >
> >  TestMetadataProvider.tables:153 expected: but was:
> >
> >  TestMetadataProvider.tablesWithTableNameFilter:212 expected: but
> was:
> >
> >  TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but
> was:
> >
> >  TestMetadataProvider.tablesWithTableFilter:176 expected: but
> was:
> >
> >
> > Tests in error:
> >
> >  TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM
> ERROR: URISyn...
> >
> >  TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth »
> UserRemote
> >
> >  TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR:
> URISyntaxExcepti...
> >
> >  TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344
> » Rpc
> >
> >  TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote
> SYSTEM E...
> >
> >  TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344
> » Rpc o...
> >
> > Thanks,
> > Tim
> >
>
>

[GitHub] drill pull request #913: - DRILL-5729 Fix Travis Build

[ilooner-mapr]

GitHub user ilooner-mapr opened a pull request:

https://github.com/apache/drill/pull/913

 - DRILL-5729 Fix Travis Build



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilooner-mapr/drill DRILL-5729

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #913


commit b52188f8091a3b22021cb1efaa42f7ebb1b2792d
Author: Timothy Farkas 
Date:   2017-08-17T23:37:24Z

 - DRILL-5729 explicitly made the travis container ubuntu precise to fix 
build errors caused by ubuntu trusty.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #912: - Test travis fix

[ilooner-mapr]

Github user ilooner-mapr closed the pull request at:

https://github.com/apache/drill/pull/912


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5729) Fix Travis Checks

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-5729:
-

 Summary: Fix Travis Checks
 Key: DRILL-5729
 URL: https://issues.apache.org/jira/browse/DRILL-5729
 Project: Apache Drill
  Issue Type: Bug
Reporter: Timothy Farkas
Assignee: Timothy Farkas
 Fix For: 1.12.0


Currently the Travis Checks are failing. The failures are happening because 
Travis recently switched their default build containers from Ubuntu precise to 
Ubuntu Trusty and we do not explicitly define the dist we build on in our 
travis.yml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #912: - Test travis fix

[ilooner-mapr]

GitHub user ilooner-mapr opened a pull request:

https://github.com/apache/drill/pull/912

 - Test travis fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilooner-mapr/drill travisFix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #912


commit 6471f93abd4175c57575a200a048d6c8a58238b0
Author: Timothy Farkas 
Date:   2017-08-17T23:21:00Z

 - Test travis fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5728) Hash Aggregate: Useless bigint value vector in the values batch

[Boaz Ben-Zvi (JIRA)]

Boaz Ben-Zvi created DRILL-5728:
---

 Summary: Hash Aggregate: Useless bigint value vector in the values 
batch
 Key: DRILL-5728
 URL: https://issues.apache.org/jira/browse/DRILL-5728
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Codegen
Affects Versions: 1.11.0
Reporter: Boaz Ben-Zvi
Priority: Minor


 When aggregating a non-nullable column (like *sum(l_partkey)* below), the code 
generation creates an extra value vector (in addition to the actual "sum" 
vector) which is used as a "nonNullCount".
   This is useless (as the underlying column is non-nullable), and wastes 
considerable memory ( 8 * 64K = 512K per each value in a batch !!)

Example query:

select sum(l_partkey) as slpk from cp.`tpch/lineitem.parquet` group by 
l_orderkry;


And as can be seen in the generated code below, the bigint value vector *vv5* 
is only used to hold a *1* flag to note "not null":


public void updateAggrValuesInternal(int incomingRowIdx, int htRowIdx)
throws SchemaChangeException
{
{
IntHolder out11 = new IntHolder();
{
out11 .value = vv8 .getAccessor().get((incomingRowIdx));
}
IntHolder in = out11;
work0 .value = vv1 .getAccessor().get((htRowIdx));
BigIntHolder value = work0;
work4 .value = vv5 .getAccessor().get((htRowIdx));
BigIntHolder nonNullCount = work4;
 
SumFunctions$IntSum_add: {
nonNullCount.value = 1;
value.value += in.value;
}
 
work0 = value;
vv1 .getMutator().set((htRowIdx), work0 .value);
work4 = nonNullCount;
vv5 .getMutator().set((htRowIdx), work4 .value);
}
}


 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill issue #911: - DRILL-5507 Made verbose info logging message debug level...

[ilooner-mapr]

Github user ilooner-mapr commented on the issue:

https://github.com/apache/drill/pull/911
  
Applied comments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Failing Tests In Master

[Padma Penumarthy]

Yes, I do see these failures.

Thanks,
Padma

> On Aug 17, 2017, at 3:55 PM, Timothy Farkas  wrote:
> 
> These tests are consistently failing for me with the latest master. Does 
> anyone else see this issue?
> 
> 
> Failed tests:
> 
>  TestMetadataProvider.tables:153 expected: but was:
> 
>  TestMetadataProvider.tablesWithTableNameFilter:212 expected: but 
> was:
> 
>  TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but 
> was:
> 
>  TestMetadataProvider.tablesWithTableFilter:176 expected: but was:
> 
> 
> Tests in error:
> 
>  TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM ERROR: 
> URISyn...
> 
>  TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth » 
> UserRemote
> 
>  TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR: 
> URISyntaxExcepti...
> 
>  TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344 » 
> Rpc
> 
>  TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote SYSTEM 
> E...
> 
>  TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344 » Rpc 
> o...
> 
> Thanks,
> Tim
> 

Failing Tests In Master

[Timothy Farkas]

These tests are consistently failing for me with the latest master. Does anyone 
else see this issue?


Failed tests:

  TestMetadataProvider.tables:153 expected: but was:

  TestMetadataProvider.tablesWithTableNameFilter:212 expected: but 
was:

  TestMetadataProvider.tablesWithSystemTableFilter:187 expected: but 
was:

  TestMetadataProvider.tablesWithTableFilter:176 expected: but was:


Tests in error:

  TestCustomUserAuthenticator.positiveUserAuth » UserRemote SYSTEM ERROR: 
URISyn...

  TestCustomUserAuthenticator.positiveUserAuthAfterNegativeUserAuth » UserRemote

  TestInfoSchema.selectFromAllTables » UserRemote SYSTEM ERROR: 
URISyntaxExcepti...

  TestViewSupport.infoSchemaWithView:350->BaseTestQuery.testRunAndReturn:344 » 
Rpc

  TestInfoSchemaFilterPushDown.testFilterPushdown_NonEqual » UserRemote SYSTEM 
E...

  TestParquetScan.testSuccessFile:58->BaseTestQuery.testRunAndReturn:344 » Rpc 
o...

Thanks,
Tim

[GitHub] drill pull request #911: - DRILL-5507 Made verbose info logging message debu...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/911#discussion_r133838104
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/BlockMapBuilder.java
 ---
@@ -228,6 +230,7 @@ public EndpointByteMap getEndpointByteMap(FileWork 
work) throws IOException {
 
 // Find submap of ranges that intersect with the rowGroup
 ImmutableRangeMap subRangeMap = 
blockMap.subRangeMap(rowGroupRange);
+Set noDrillbitHosts = Sets.newHashSet();
--- End diff --

Consider `final Set noDrillbitHosts = logger.isDebugEnabled() ? 
Sets.newHashSet() : null;`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #911: - DRILL-5507 Made verbose info logging message debu...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/911#discussion_r133838389
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/BlockMapBuilder.java
 ---
@@ -246,12 +249,16 @@ public EndpointByteMap getEndpointByteMap(FileWork 
work) throws IOException {
 DrillbitEndpoint endpoint = getDrillBitEndpoint(host);
 if (endpoint != null) {
   endpointByteMap.add(endpoint, bytes);
-} else {
-  logger.info("Failure finding Drillbit running on host {}.  
Skipping affinity to that host.", host);
+} else if (logger.isDebugEnabled()) {
--- End diff --

and `} else if (noDrillbitHosts != null && noDrillbitHosts.add(host)) { 
logger.debug(...); }`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5727) Update release profile to generate SHA-512 checksum.

[Parth Chandra (JIRA)]

Parth Chandra created DRILL-5727:


 Summary: Update release profile to generate SHA-512 checksum.
 Key: DRILL-5727
 URL: https://issues.apache.org/jira/browse/DRILL-5727
 Project: Apache Drill
  Issue Type: Bug
Reporter: Parth Chandra


Per latest release guidelines, we should generate a sha-512 checksum with the 
release artifacts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill issue #911: - DRILL-5507 Made verbose info logging message debug level...

[ilooner-mapr]

Github user ilooner-mapr commented on the issue:

https://github.com/apache/drill/pull/911
  
@paul-rogers 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #911: - DRILL-5507 Made verbose info logging message debu...

[ilooner-mapr]

GitHub user ilooner-mapr opened a pull request:

https://github.com/apache/drill/pull/911

 - DRILL-5507 Made verbose info logging message debug level and print…

…ed it less frequently

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilooner-mapr/drill DRILL-5507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/911.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #911


commit 34d5fe6e215526176a876fd076b6110ffa8c829d
Author: Timothy Farkas 
Date:   2017-08-17T21:29:38Z

 - DRILL-5507 Made verbose info logging message debug level and printed it 
less frequently




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #910: DRILL-5726: Support Impersonation without authentic...

[arina-ielchiieva]

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/910

DRILL-5726: Support Impersonation without authentication for REST API

Details in [DRILL-5726](https://issues.apache.org/jira/browse/DRILL-5726).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-5726

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #910


commit f3986726f601ed31f93209b2daec497a7fbd2870
Author: Arina Ielchiieva 
Date:   2017-08-17T12:08:12Z

DRILL-5726: Support Impersonation without authentication for REST API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5726) Support Impersonation without authentication for REST API

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-5726:
---

 Summary: Support Impersonation without authentication for REST API
 Key: DRILL-5726
 URL: https://issues.apache.org/jira/browse/DRILL-5726
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.11.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
 Fix For: 1.12.0


Today if a user is not authenticated via REST API then there is no way to 
provide a user name for executing queries. It will by default be executed as 
"anonymous" user. This doesn't work when impersonation without authentication 
is enabled on Drill server side, since anonymous user doesn't exist the query 
will fail. We need a way to provide a user name when impersonation is enabled 
on Drill side and query is executed from REST API.

_Implementation details:_
When only impersonation is enabled form-based authentication will be used.
On Web UI user will be prompted to enter only login, then session for that user 
will be created, user will be treated as admin. Form-based authentication will 
cache user information, so user won't need to set username each time he / she 
wants to execute the query. Log in / out options will be also available. 
Screenshot of login page is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #909: DRILL-4264: Allow field names to include dots

[vvysotskyi]

GitHub user vvysotskyi opened a pull request:

https://github.com/apache/drill/pull/909

DRILL-4264: Allow field names to include dots

1. Removed checking the field name for dots.
2. Replaced using `SchemaPath.getAsUnescapedPath()` method by 
`SchemaPath.getRootSegmentPath()` and 
`SchemaPathUtil.getMaterializedFieldFromSchemaPath()` where it is needed. 
3. Replaced using `MaterializedField.getPath()` and 
`MaterializedField.getLastName()` methods by `MaterializedField.getName()` 
method and checked the correctness of the behaviour.
4. Added tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vvysotskyi/drill DRILL-4264

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #909


commit 4ba59488a96fb79455b192ed960a728481ceaf93
Author: Volodymyr Vysotskyi 
Date:   2017-07-05T19:08:59Z

DRILL-4264: Allow field names to include dots




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #908: DRILL-5725: Update Jackson version to 2.7.8

[vvysotskyi]

GitHub user vvysotskyi opened a pull request:

https://github.com/apache/drill/pull/908

DRILL-5725: Update Jackson version to 2.7.8



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vvysotskyi/drill DRILL-5725

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/908.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #908


commit 283d1c73cbc2a72aa55396dc11a221b9380f091d
Author: Volodymyr Vysotskyi 
Date:   2017-08-16T14:16:55Z

DRILL-5725: Update Jackson version to 2.7.8




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5725) Update Jackson version to 2.7.8

[Volodymyr Vysotskyi (JIRA)]

Volodymyr Vysotskyi created DRILL-5725:
--

 Summary: Update Jackson version to 2.7.8
 Key: DRILL-5725
 URL: https://issues.apache.org/jira/browse/DRILL-5725
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Volodymyr Vysotskyi


Currently, Drill uses Jackson 2.7.1. The goal of this Jira is to update Jackson 
version to 2.7.8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)