[jira] [Commented] (DRILL-5132) Context based dynamic parameterization of views
[
https://issues.apache.org/jira/browse/DRILL-5132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772148#comment-15772148
]
ASF GitHub Bot commented on DRILL-5132:
---
Github user nagarajanchinnasamy commented on the issue:
https://github.com/apache/drill/pull/685
@arina-ielchiieva @sudheeshkatkam as
[DRILL-5132](https://issues.apache.org/jira/browse/DRILL-5132) is being
explored to be addressed through UDFs... I am reverting back to original design
as suggested by @arina-ielchiieva .
Will submit a new commit with original design shortly. Sorry about the
detour.
> Context based dynamic parameterization of views
> ---
>
> Key: DRILL-5132
> URL: https://issues.apache.org/jira/browse/DRILL-5132
> Project: Apache Drill
> Issue Type: Wish
> Components: Server
>Reporter: Nagarajan Chinnasamy
>Priority: Critical
> Labels: authentication, context, isolation, jdbcstorage,
> multi-tenancy, session-context, session-parameter
>
> *Requirement*
> Its known that Views in SQL cannot have custom dynamic parameters/variables.
> Please refer to [Justin
> Swanhart|http://stackoverflow.com/users/679236/justin-swanhart]'s response to
> [this SO
> question|http://stackoverflow.com/questions/2281890/can-i-create-view-with-parameter-in-mysql]
> in handling dynamic parameterization of views.
> [The PR #685|https://github.com/apache/drill/pull/685]
> [DRILL-5043|https://issues.apache.org/jira/browse/DRILL-5043?filter=-2]
> originated based on this requirement so that we could build views that can
> dynamically filter records based on some dynamic values (like current
> tenant-id, user role etc.)
> *Since Drill's basic unit is a View... having such built-in support can bring
> in dynamism into the whole game.*
> This feature can be utilized for:
> * *Data Isolation in Shared Multi-Tenant environments* based on Custom Tenant
> Discriminator Column
> * *Data Protection in building Chained Views* with Custom Dynamic Filters
> To explain this further, If we assume that:
> # As and when the user connection is established, we populate session context
> with session parameters such as:
> #* Tenant ID of the currently logged in user
> #* Roles of the currently logged in user
> # We expose the session context information through context-based-functions
> such as:
> #* *session_id* -- that returns unique id of the session
> #* *session_parameter('')* - that returns the value of the
> session parameter
> then a view created with the following kind of query:
> {code}
> create or replace view dynamic_filter_view as select
>a.field as a_field
>b.field as b_field
> from
>a_table as a
> left join
>b_table as b
> on
>a.bId = b.Id
> where
>session_parameter('tenantId')=a.tenantId
> {code}
> becomes a query that has built-in support for dynamic parameterization that
> only returns records of the tenant of the currently logged in user. This is a
> very useful feature in a shared-multi-tenant environment where data is
> isolated using multi-tenant-descriminator column 'tenantId'.
> When building chained views this feature will be useful in filtering records
> based on context based parameters.
> This feature will particularly be useful for data isolation / data protection
> with *jdbc storage plugins* where drill-authenticated-credentials are not
> passed to jdbc connection authentication. A jdbc storage has hard-coded,
> shared credentials. Hence the the responsibility of data isolation / data
> protection lies with Views themselves. Hence, the need for built-in support
> of context based dynamic parameters in Views.
> *Design/Implementation Considerations:*
> * Session parameters can be obtained through authenticators so that custom
> authenticators can return a HashMap of parameters obtained from external
> systems.
> * Introduce SessionContext to hold sessionId and sessionParameters
> * Implement context based functions session_id and session_parameter()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
[
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771939#comment-15771939
]
Paul Rogers commented on DRILL-5157:
For the "managed" external sort branch, the problem is avoided by replacing the
(unnecessary) Parquet scan with a mock table scan. The scan is needed only so
it can be interrupted, and the mock scan works just as well.
> TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async
> Snappy error
> ---
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall
> suite. The error does not appear if the test is run individually in the
> debugger.
> Stack trace that seems to show that the code was trying to find a Snappy
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
> at org.xerial.snappy.Snappy.(Snappy.java:44)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
> at
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}
> Resulting memory leak if the test is allowed to complete:
> {code}
> java.lang.AssertionError: Query state is incorrect (expected: CANCELED,
> actual: FAILED) AND/OR
> Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan]
> closed with outstanding buffers allocated (2).
> Allocator(op:3:0:15:ParquetRowGroupScan) 100/393216/3162112/100
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 131072, references: 1, life: 73148192887288..0, allocatorManager:
> [3027, life: 73148192235794..0] holds 1 buffers.
> DrillBuf[4949], udle: [3028 0..131072]
> ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 262144, references: 1, life: 73148451288840..0, allocatorManager:
> [3091, life: 73148451257480..0] holds 1 buffers.
> DrillBuf[5017], udle: [3092 0..262144]
> reservations: 0
> Fragment 3:0
> [Error Id: 8502074b-f488-4a14-bf7d-a2a4480392cd on 172.30.1.67:31016]
> at
> org.apache.drill.exec.server.TestDrillbitResilience.assertStateCompleted(TestDrillbitResilience.java:861)
> at
> org.apache.drill.exec.server.TestDrillbitResilience.assertCancelledWithoutException(TestDrillbitResilience.java:876)
> at
>
[jira] [Commented] (DRILL-5155) TestDrillbitResilience unit test is not resilient
[
https://issues.apache.org/jira/browse/DRILL-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771926#comment-15771926
]
Jacques Nadeau commented on DRILL-5155:
---
+1 on the jira title. Made me laugh :)
> TestDrillbitResilience unit test is not resilient
> -
>
> Key: DRILL-5155
> URL: https://issues.apache.org/jira/browse/DRILL-5155
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The unit test {{TestDrillbitResilience}} plays quite rough with a set of
> Drillbits, forcing a number of error conditions to see if the Drillbits can
> recover. The test cases are good, but they interact with each other to make
> the test as a whole quite fragile. The failure of any one test tends to cause
> others to fail. When tests are run individually, they may run. But, when run
> as a suite, they fail due to cross-interactions.
> Restructure the test to make the tests more independent so that one test does
> not change the state of the cluster expected by a different test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
[
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771786#comment-15771786
]
Paul Rogers edited comment on DRILL-5157 at 12/23/16 3:50 AM:
--
There are really three errors here.
First is that the Snappy library is not available on the Mac where I ran the
tests. See:
http://stackoverflow.com/questions/15158405/unable-to-load-native-hadoop-library-for-mac-os-x
It seems that the Parquet file being read was encoded with Snappy, and so the
Snappy library is required.
The second error is that we do not fall back to a Java implementation when the
native library is unavailable. (See following note.)
The third error is that the error caused a memory leak. In general, we'd like
errors, even those like the above, to clean up memory to avoid putting the
Drillbit into a bad state.
was (Author: paul-rogers):
There are really two errors here.
First is that the Snappy library is not available on the Mac where I ran the
tests. See:
http://stackoverflow.com/questions/15158405/unable-to-load-native-hadoop-library-for-mac-os-x
It seems that the Parquet file being read was encoded with Snappy, and so the
Snappy library is required.
The solution to this is to describe how to obtain the library, or find some
Java alternative when Snappy is not available.
The second error is that the error caused a memory leak. In general, we'd like
errors, even those like the above, to clean up memory to avoid putting the
Drillbit into a bad state.
> TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async
> Snappy error
> ---
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall
> suite. The error does not appear if the test is run individually in the
> debugger.
> Stack trace that seems to show that the code was trying to find a Snappy
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
> at org.xerial.snappy.Snappy.(Snappy.java:44)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
> at
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}
> Resulting memory leak if the test is allowed to complete:
> {code}
> java.lang.AssertionError: Query state is incorrect (expected: CANCELED,
> actual: FAILED) AND/OR
> Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM ERROR: IllegalStateException:
[jira] [Comment Edited] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
[
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771789#comment-15771789
]
Paul Rogers edited comment on DRILL-5157 at 12/23/16 3:49 AM:
--
If the native library is not available, a common Hadoop practice (for the HDFS
file system) is to fall back to a slower Java implementation to allow the code
to execute. The following is a Java implementation of Snappy:
https://code.google.com/archive/p/jsnappy/
Another: https://github.com/dain/snappy
Both are from the Snappy home page: https://google.github.io/snappy/
was (Author: paul-rogers):
If the native library is not available, a common Hadoop practice (for the HDFS
file system) is to fall back to a slower Java implementation to allow the code
to execute. The following is a Java implementation of Snappy:
https://code.google.com/archive/p/jsnappy/
> TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async
> Snappy error
> ---
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall
> suite. The error does not appear if the test is run individually in the
> debugger.
> Stack trace that seems to show that the code was trying to find a Snappy
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
> at org.xerial.snappy.Snappy.(Snappy.java:44)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
> at
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}
> Resulting memory leak if the test is allowed to complete:
> {code}
> java.lang.AssertionError: Query state is incorrect (expected: CANCELED,
> actual: FAILED) AND/OR
> Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan]
> closed with outstanding buffers allocated (2).
> Allocator(op:3:0:15:ParquetRowGroupScan) 100/393216/3162112/100
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 131072, references: 1, life: 73148192887288..0, allocatorManager:
> [3027, life: 73148192235794..0] holds 1 buffers.
> DrillBuf[4949], udle: [3028 0..131072]
> ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 262144, references: 1, life: 73148451288840..0, allocatorManager:
> [3091, life:
[jira] [Commented] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
[
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771789#comment-15771789
]
Paul Rogers commented on DRILL-5157:
If the native library is not available, a common Hadoop practice (for the HDFS
file system) is to fall back to a slower Java implementation to allow the code
to execute. The following is a Java implementation of Snappy:
https://code.google.com/archive/p/jsnappy/
> TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async
> Snappy error
> ---
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall
> suite. The error does not appear if the test is run individually in the
> debugger.
> Stack trace that seems to show that the code was trying to find a Snappy
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
> at org.xerial.snappy.Snappy.(Snappy.java:44)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
> at
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}
> Resulting memory leak if the test is allowed to complete:
> {code}
> java.lang.AssertionError: Query state is incorrect (expected: CANCELED,
> actual: FAILED) AND/OR
> Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan]
> closed with outstanding buffers allocated (2).
> Allocator(op:3:0:15:ParquetRowGroupScan) 100/393216/3162112/100
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 131072, references: 1, life: 73148192887288..0, allocatorManager:
> [3027, life: 73148192235794..0] holds 1 buffers.
> DrillBuf[4949], udle: [3028 0..131072]
> ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 262144, references: 1, life: 73148451288840..0, allocatorManager:
> [3091, life: 73148451257480..0] holds 1 buffers.
> DrillBuf[5017], udle: [3092 0..262144]
> reservations: 0
> Fragment 3:0
> [Error Id: 8502074b-f488-4a14-bf7d-a2a4480392cd on 172.30.1.67:31016]
> at
> org.apache.drill.exec.server.TestDrillbitResilience.assertStateCompleted(TestDrillbitResilience.java:861)
> at
> org.apache.drill.exec.server.TestDrillbitResilience.assertCancelledWithoutException(TestDrillbitResilience.java:876)
> at
>
[jira] [Commented] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
[
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771786#comment-15771786
]
Paul Rogers commented on DRILL-5157:
There are really two errors here.
First is that the Snappy library is not available on the Mac where I ran the
tests. See:
http://stackoverflow.com/questions/15158405/unable-to-load-native-hadoop-library-for-mac-os-x
It seems that the Parquet file being read was encoded with Snappy, and so the
Snappy library is required.
The solution to this is to describe how to obtain the library, or find some
Java alternative when Snappy is not available.
The second error is that the error caused a memory leak. In general, we'd like
errors, even those like the above, to clean up memory to avoid putting the
Drillbit into a bad state.
> TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async
> Snappy error
> ---
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Paul Rogers
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
> fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall
> suite. The error does not appear if the test is run individually in the
> debugger.
> Stack trace that seems to show that the code was trying to find a Snappy
> native library:
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
> at org.xerial.snappy.Snappy.(Snappy.java:44)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
> at
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
> at
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}
> Resulting memory leak if the test is allowed to complete:
> {code}
> java.lang.AssertionError: Query state is incorrect (expected: CANCELED,
> actual: FAILED) AND/OR
> Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan]
> closed with outstanding buffers allocated (2).
> Allocator(op:3:0:15:ParquetRowGroupScan) 100/393216/3162112/100
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 131072, references: 1, life: 73148192887288..0, allocatorManager:
> [3027, life: 73148192235794..0] holds 1 buffers.
> DrillBuf[4949], udle: [3028 0..131072]
> ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
> size: 262144, references: 1, life: 73148451288840..0, allocatorManager:
> [3091, life: 73148451257480..0] holds 1 buffers.
> DrillBuf[5017], udle: [3092 0..262144]
>
[jira] [Commented] (DRILL-5156) Bit-Client thread finds closed allocator in TestDrillbitResilience unit test
[
https://issues.apache.org/jira/browse/DRILL-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771772#comment-15771772
]
Paul Rogers commented on DRILL-5156:
Continuing to investigate, it appears that the RPC threads hang around after
shutting down the Drillbit if the debugger is stopped at the exception
breakpoint. In particular, I ran the full test, with the
{{IllegalStateException}} breakpoint. The breakpoint was hit in one tests. That
test shuts down its Drillbit at the end. Then, another test started that
created a new Drillbit. It seams that the first Drillbit did not wait for the
RPC threads to exit; leaving orphaned threads (those stopped in the debugger.)
Seems the Drillbit should refuse to exit until all child threads have completed.
> Bit-Client thread finds closed allocator in TestDrillbitResilience unit test
>
>
> Key: DRILL-5156
> URL: https://issues.apache.org/jira/browse/DRILL-5156
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Priority: Minor
>
> RPC thread attempts to access a closed allocator during the
> {{TestDrillbitResilience}} unit test.
> Set a Java exception breakpoint for {{IllegalStateException}}. Run the
> {{TestDrillbitResilience}} unit tests.
> You will see quite a few exceptions, including the following in a thread
> called BitClient-1:
> {code}
> RootAllocator(BaseAllocator).assertOpen() line 109
> RootAllocator(BaseAllocator).buffer(int) line 191
> DrillByteBufAllocator.buffer(int) line 49
> DrillByteBufAllocator.ioBuffer(int) line 64
> AdaptiveRecvByteBufAllocatpr$HandleImpl.allocate(ByteBufAllocator) line 104
> NioSocketChannel$NioSocketChannelUnsafe(...).read() line 117
> ...
> NioEventLoop.run() line 354
> {code}
> The test continues (then fails for some other reason), which is why this is
> marked as minor. Still, it seems odd that the client thread should attempt to
> access a closed allocator.
> At this point, it is not clear how we got into this state. The test itself is
> waiting for a response from the server in the {{tailsAfterMSorterSorting}}
> test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
[
https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5157:
---
Description:
Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
fails with the following stack trace and the memory leak trace shown second.
Strangely, this error appears only if the test is run as part of the overall
suite. The error does not appear if the test is run individually in the
debugger.
Stack trace that seems to show that the code was trying to find a Snappy native
library:
{code}
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
at org.xerial.snappy.Snappy.(Snappy.java:44)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
at
org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
…
{code}
Resulting memory leak if the test is allowed to complete:
{code}
java.lang.AssertionError: Query state is incorrect (expected: CANCELED, actual:
FAILED) AND/OR
Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan]
closed with outstanding buffers allocated (2).
Allocator(op:3:0:15:ParquetRowGroupScan) 100/393216/3162112/100
(res/actual/peak/limit)
child allocators: 0
ledgers: 2
ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
size: 131072, references: 1, life: 73148192887288..0, allocatorManager: [3027,
life: 73148192235794..0] holds 1 buffers.
DrillBuf[4949], udle: [3028 0..131072]
ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
size: 262144, references: 1, life: 73148451288840..0, allocatorManager: [3091,
life: 73148451257480..0] holds 1 buffers.
DrillBuf[5017], udle: [3092 0..262144]
reservations: 0
Fragment 3:0
[Error Id: 8502074b-f488-4a14-bf7d-a2a4480392cd on 172.30.1.67:31016]
at
org.apache.drill.exec.server.TestDrillbitResilience.assertStateCompleted(TestDrillbitResilience.java:861)
at
org.apache.drill.exec.server.TestDrillbitResilience.assertCancelledWithoutException(TestDrillbitResilience.java:876)
at
org.apache.drill.exec.server.TestDrillbitResilience.doMemoryLeaksWhenCancelled(TestDrillbitResilience.java:680)
at
org.apache.drill.exec.server.TestDrillbitResilience.memoryLeaksWhenCancelled(TestDrillbitResilience.java:647)
...
{code}
was:
Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
fails with the following stack trace and the memory leak trace shown second.
Stack trace that seems to show that the code was trying to find a Snappy native
library:
{code}
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
[jira] [Created] (DRILL-5157) TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
Paul Rogers created DRILL-5157:
--
Summary: TestDrillbitResilience.doMemoryLeaksWhenCancelled fails
with Parquet async Snappy error
Key: DRILL-5157
URL: https://issues.apache.org/jira/browse/DRILL-5157
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Paul Rogers
Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It
fails with the following stack trace and the memory leak trace shown second.
Stack trace that seems to show that the code was trying to find a Snappy native
library:
{code}
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
at org.xerial.snappy.Snappy.(Snappy.java:44)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
at
org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
…
{code}
Resulting memory leak if the test is allowed to complete:
{code}
java.lang.AssertionError: Query state is incorrect (expected: CANCELED, actual:
FAILED) AND/OR
Exception thrown: org.apache.drill.common.exceptions.UserRemoteException:
SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan]
closed with outstanding buffers allocated (2).
Allocator(op:3:0:15:ParquetRowGroupScan) 100/393216/3162112/100
(res/actual/peak/limit)
child allocators: 0
ledgers: 2
ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
size: 131072, references: 1, life: 73148192887288..0, allocatorManager: [3027,
life: 73148192235794..0] holds 1 buffers.
DrillBuf[4949], udle: [3028 0..131072]
ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true,
size: 262144, references: 1, life: 73148451288840..0, allocatorManager: [3091,
life: 73148451257480..0] holds 1 buffers.
DrillBuf[5017], udle: [3092 0..262144]
reservations: 0
Fragment 3:0
[Error Id: 8502074b-f488-4a14-bf7d-a2a4480392cd on 172.30.1.67:31016]
at
org.apache.drill.exec.server.TestDrillbitResilience.assertStateCompleted(TestDrillbitResilience.java:861)
at
org.apache.drill.exec.server.TestDrillbitResilience.assertCancelledWithoutException(TestDrillbitResilience.java:876)
at
org.apache.drill.exec.server.TestDrillbitResilience.doMemoryLeaksWhenCancelled(TestDrillbitResilience.java:680)
at
org.apache.drill.exec.server.TestDrillbitResilience.memoryLeaksWhenCancelled(TestDrillbitResilience.java:647)
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5156) Bit-Client thread finds closed allocator in TestDrillbitResilience unit test
Paul Rogers created DRILL-5156:
--
Summary: Bit-Client thread finds closed allocator in
TestDrillbitResilience unit test
Key: DRILL-5156
URL: https://issues.apache.org/jira/browse/DRILL-5156
Project: Apache Drill
Issue Type: Bug
Reporter: Paul Rogers
Priority: Minor
RPC thread attempts to access a closed allocator during the
{{TestDrillbitResilience}} unit test.
Set a Java exception breakpoint for {{IllegalStateException}}. Run the
{{TestDrillbitResilience}} unit tests.
You will see quite a few exceptions, including the following in a thread called
BitClient-1:
{code}
RootAllocator(BaseAllocator).assertOpen() line 109
RootAllocator(BaseAllocator).buffer(int) line 191
DrillByteBufAllocator.buffer(int) line 49
DrillByteBufAllocator.ioBuffer(int) line 64
AdaptiveRecvByteBufAllocatpr$HandleImpl.allocate(ByteBufAllocator) line 104
NioSocketChannel$NioSocketChannelUnsafe(...).read() line 117
...
NioEventLoop.run() line 354
{code}
The test continues (then fails for some other reason), which is why this is
marked as minor. Still, it seems odd that the client thread should attempt to
access a closed allocator.
At this point, it is not clear how we got into this state. The test itself is
waiting for a response from the server in the {{tailsAfterMSorterSorting}} test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5155) TestDrillbitResilience unit test is not resilient
Paul Rogers created DRILL-5155:
--
Summary: TestDrillbitResilience unit test is not resilient
Key: DRILL-5155
URL: https://issues.apache.org/jira/browse/DRILL-5155
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Priority: Minor
The unit test {{TestDrillbitResilience}} plays quite rough with a set of
Drillbits, forcing a number of error conditions to see if the Drillbits can
recover. The test cases are good, but they interact with each other to make the
test as a whole quite fragile. The failure of any one test tends to cause
others to fail. When tests are run individually, they may run. But, when run as
a suite, they fail due to cross-interactions.
Restructure the test to make the tests more independent so that one test does
not change the state of the cluster expected by a different test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5137) Optimize count(*) queries on MapR-DB Binary Tables
[
https://issues.apache.org/jira/browse/DRILL-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771347#comment-15771347
]
ASF GitHub Bot commented on DRILL-5137:
---
Github user spanchamiamapr commented on a diff in the pull request:
https://github.com/apache/drill/pull/700#discussion_r93710708
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
---
@@ -124,6 +124,11 @@ public HBaseRecordReader(Connection connection,
HBaseSubScan.HBaseSubScanSpec su
} else {
rowKeyOnly = false;
transformed.add(ROW_KEY_PATH);
+ /* DRILL-5137 - optimize count(*) queries on MapR-DB Binary tables */
+ if (isSkipQuery()) {
--- End diff --
I thought of this, however I got stuck because there is no
vectorContainerWriter in HBaseRecordReader class. If you can send a diff of how
to do this, I can include that too. Thanks!
> Optimize count(*) queries on MapR-DB Binary Tables
> --
>
> Key: DRILL-5137
> URL: https://issues.apache.org/jira/browse/DRILL-5137
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HBase, Storage - MapRDB
>Affects Versions: 1.9.0
>Reporter: Abhishek Girish
>Assignee: Smidth Panchamia
>
> This is related to DRILL-5065, but applies to MapR-DB Binary tables
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5137) Optimize count(*) queries on MapR-DB Binary Tables
[
https://issues.apache.org/jira/browse/DRILL-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771321#comment-15771321
]
ASF GitHub Bot commented on DRILL-5137:
---
Github user spanchamiamapr commented on a diff in the pull request:
https://github.com/apache/drill/pull/700#discussion_r93709204
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
---
@@ -124,6 +124,11 @@ public HBaseRecordReader(Connection connection,
HBaseSubScan.HBaseSubScanSpec su
} else {
rowKeyOnly = false;
transformed.add(ROW_KEY_PATH);
+ /* DRILL-5137 - optimize count(*) queries on MapR-DB Binary tables */
--- End diff --
Yes, I verified this in my testing.
> Optimize count(*) queries on MapR-DB Binary Tables
> --
>
> Key: DRILL-5137
> URL: https://issues.apache.org/jira/browse/DRILL-5137
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HBase, Storage - MapRDB
>Affects Versions: 1.9.0
>Reporter: Abhishek Girish
>Assignee: Smidth Panchamia
>
> This is related to DRILL-5065, but applies to MapR-DB Binary tables
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-5154) OOM error in external sort on top of 400GB data set generated using terasort benchamark
[
https://issues.apache.org/jira/browse/DRILL-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Challapalli updated DRILL-5154:
-
Attachment: 27a3de95-e30b-8890-6653-80fd6c49a3a1.sys.drill
The log file is too big to attach to a jira. Reach out to me if further
information is needed
> OOM error in external sort on top of 400GB data set generated using terasort
> benchamark
> ---
>
> Key: DRILL-5154
> URL: https://issues.apache.org/jira/browse/DRILL-5154
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.10.0
>Reporter: Rahul Challapalli
> Attachments: 27a3de95-e30b-8890-6653-80fd6c49a3a1.sys.drill
>
>
> git.commit.id.abbrev=cf2b7c7
> The below query fails with an OOM in external sort
> {code}
> No of drillbits : 1
> Nodes in Mapr cluster : 2
> DRILL_MAX_DIRECT_MEMORY="16G"
> DRILL_MAX_HEAP="4G"
> select * from (select * from
> dfs.`/drill/testdata/resource-manager/terasort-data/part-m-0.tbl` order
> by columns[0]) d where d.columns[0] = 'null';
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing
> the query.
> Unable to allocate buffer of size 8388608 due to memory limit. Current
> allocation: 8441872
> Fragment 1:6
> [Error Id: 87ede736-b480-4286-b472-7694fdd2f7da on qa-node183.qa.lab:31010]
> (state=,code=0)
> {code}
> I attached the logs and the query profile
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5154) OOM error in external sort on top of 400GB data set generated using terasort benchamark
[
https://issues.apache.org/jira/browse/DRILL-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770892#comment-15770892
]
Rahul Challapalli commented on DRILL-5154:
--
Stacktrace from the logs
{code}
2016-12-22 11:13:09,179 [27a3de95-e30b-8890-6653-80fd6c49a3a1:frag:1:6] INFO
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
ran out of memory while executing the query. (Unable to allocate buffer of size
8388608 due to memory limit. Current allocation: 8441872)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Unable to allocate buffer of size 8388608 due to memory limit. Current
allocation: 8441872
[Error Id: 87ede736-b480-4286-b472-7694fdd2f7da ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:242)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_92]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_92]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 8388608 due to memory limit. Current allocation: 8441872
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.VarCharVector.reAlloc(VarCharVector.java:401)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:556)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector$Mutator.addSafe(RepeatedVarCharVector.java:367)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector$Mutator.addSafe(RepeatedVarCharVector.java:362)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector.copyFromSafe(RepeatedVarCharVector.java:224)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.PriorityQueueCopierGen90.doCopy(PriorityQueueCopierTemplate.java:25)
~[na:na]
at
org.apache.drill.exec.test.generated.PriorityQueueCopierGen90.next(PriorityQueueCopierTemplate.java:75)
~[na:na]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:562)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:414)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:92)
[jira] [Commented] (DRILL-5154) OOM error in external sort on top of 400GB data set generated using terasort benchamark
[
https://issues.apache.org/jira/browse/DRILL-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770891#comment-15770891
]
Rahul Challapalli commented on DRILL-5154:
--
Stacktrace from the logs
{code}
2016-12-22 11:13:09,179 [27a3de95-e30b-8890-6653-80fd6c49a3a1:frag:1:6] INFO
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
ran out of memory while executing the query. (Unable to allocate buffer of size
8388608 due to memory limit. Current allocation: 8441872)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Unable to allocate buffer of size 8388608 due to memory limit. Current
allocation: 8441872
[Error Id: 87ede736-b480-4286-b472-7694fdd2f7da ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:242)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_92]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_92]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 8388608 due to memory limit. Current allocation: 8441872
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.VarCharVector.reAlloc(VarCharVector.java:401)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:556)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector$Mutator.addSafe(RepeatedVarCharVector.java:367)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector$Mutator.addSafe(RepeatedVarCharVector.java:362)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector.copyFromSafe(RepeatedVarCharVector.java:224)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.PriorityQueueCopierGen90.doCopy(PriorityQueueCopierTemplate.java:25)
~[na:na]
at
org.apache.drill.exec.test.generated.PriorityQueueCopierGen90.next(PriorityQueueCopierTemplate.java:75)
~[na:na]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:562)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:414)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:92)
[jira] [Created] (DRILL-5154) OOM error in external sort on top of 400GB data set generated using terasort benchamark
Rahul Challapalli created DRILL-5154:
Summary: OOM error in external sort on top of 400GB data set
generated using terasort benchamark
Key: DRILL-5154
URL: https://issues.apache.org/jira/browse/DRILL-5154
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.10.0
Reporter: Rahul Challapalli
git.commit.id.abbrev=cf2b7c7
The below query fails with an OOM in external sort
{code}
No of drillbits : 1
Nodes in Mapr cluster : 2
DRILL_MAX_DIRECT_MEMORY="16G"
DRILL_MAX_HEAP="4G"
select * from (select * from
dfs.`/drill/testdata/resource-manager/terasort-data/part-m-0.tbl` order by
columns[0]) d where d.columns[0] = 'null';
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the
query.
Unable to allocate buffer of size 8388608 due to memory limit. Current
allocation: 8441872
Fragment 1:6
[Error Id: 87ede736-b480-4286-b472-7694fdd2f7da on qa-node183.qa.lab:31010]
(state=,code=0)
{code}
I attached the logs and the query profile
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5068) Add a new system table for completed profiles
[ https://issues.apache.org/jira/browse/DRILL-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769982#comment-15769982 ] Hongze Zhang commented on DRILL-5068: - Hi Khurram, Thank you for your guide! I will mail to the dev list to ask the question. > Add a new system table for completed profiles > - > > Key: DRILL-5068 > URL: https://issues.apache.org/jira/browse/DRILL-5068 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Information Schema >Affects Versions: 1.8.0 > Environment: Fedora 25 > OpenJDK 8 > Firefox 50.0 >Reporter: Hongze Zhang >Assignee: Hongze Zhang > Fix For: Future > > > Hi, > Currently the profile page on UI is still not detailed enough for some > complicated uses (eg. show all failed queries during these three days), we > can only access latest 100 query profiles on this page. > We may sometimes need a specific system table for querying completed profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5068) Add a new system table for completed profiles
[ https://issues.apache.org/jira/browse/DRILL-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769928#comment-15769928 ] Khurram Faraaz commented on DRILL-5068: --- [~zbdzzg] you should ask your question (Is this thing useful for next version of Drill ?) on Drill dev list. > Add a new system table for completed profiles > - > > Key: DRILL-5068 > URL: https://issues.apache.org/jira/browse/DRILL-5068 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Information Schema >Affects Versions: 1.8.0 > Environment: Fedora 25 > OpenJDK 8 > Firefox 50.0 >Reporter: Hongze Zhang >Assignee: Hongze Zhang > Fix For: Future > > > Hi, > Currently the profile page on UI is still not detailed enough for some > complicated uses (eg. show all failed queries during these three days), we > can only access latest 100 query profiles on this page. > We may sometimes need a specific system table for querying completed profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-5039) NPE - CTAS PARTITION BY ()
[
https://issues.apache.org/jira/browse/DRILL-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-5039:
Fix Version/s: Future
> NPE - CTAS PARTITION BY ()
>
>
> Key: DRILL-5039
> URL: https://issues.apache.org/jira/browse/DRILL-5039
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: Future
>
>
> We are seeing an NPE, when CTAS is used with PARTITION BY
> () and all columns are projected in SELECT of CTAS.
> Drill 1.9.0
> git commit ID : db30854
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl
> . . . . . . . . . . . . . . > PARTITION BY (col_chr)
> . . . . . . . . . . . . . . > AS
> . . . . . . . . . . . . . . > SELECT col_int, col_chr, col_vrchr1, col_vrchr2
> , col_dt, col_tim, col_tmstmp , col_flt, col_intrvl_yr , col_intrvl_day ,
> col_bln
> . . . . . . . . . . . . . . > FROM typeall_l;
> Error: SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> (state=,code=0)
> {noformat}
> {noformat}
> 2016-11-12 19:54:14,901 [27d88c99-a64d-0317-ba3b-d78195cf85cc:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> ~[drill-common-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.9.0.jar:1.9.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
> ~[vector-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.test.generated.ProjectorGen3.doEval(ProjectorTemplate.java:88)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.ProjectorGen3.projectRecords(ProjectorTemplate.java:62)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>
[jira] [Assigned] (DRILL-5039) NPE - CTAS PARTITION BY ()
[
https://issues.apache.org/jira/browse/DRILL-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-5039:
---
Assignee: Arina Ielchiieva
> NPE - CTAS PARTITION BY ()
>
>
> Key: DRILL-5039
> URL: https://issues.apache.org/jira/browse/DRILL-5039
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
>
> We are seeing an NPE, when CTAS is used with PARTITION BY
> () and all columns are projected in SELECT of CTAS.
> Drill 1.9.0
> git commit ID : db30854
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl
> . . . . . . . . . . . . . . > PARTITION BY (col_chr)
> . . . . . . . . . . . . . . > AS
> . . . . . . . . . . . . . . > SELECT col_int, col_chr, col_vrchr1, col_vrchr2
> , col_dt, col_tim, col_tmstmp , col_flt, col_intrvl_yr , col_intrvl_day ,
> col_bln
> . . . . . . . . . . . . . . > FROM typeall_l;
> Error: SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> (state=,code=0)
> {noformat}
> {noformat}
> 2016-11-12 19:54:14,901 [27d88c99-a64d-0317-ba3b-d78195cf85cc:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 0:0
> [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> ~[drill-common-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.9.0.jar:1.9.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
> ~[vector-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.test.generated.ProjectorGen3.doEval(ProjectorTemplate.java:88)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.ProjectorGen3.projectRecords(ProjectorTemplate.java:62)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
>
[jira] [Updated] (DRILL-5117) Compile error when query a json file with 1000+columns
[
https://issues.apache.org/jira/browse/DRILL-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-5117:
--
Attachment: manyColsInJson.json
> Compile error when query a json file with 1000+columns
> --
>
> Key: DRILL-5117
> URL: https://issues.apache.org/jira/browse/DRILL-5117
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
> Attachments: manyColsInJson.json
>
>
> Query failed with compile error when we querying a json file with
> 1000+columns:
> {noformat}
> 0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
> Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class
> "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
> (state=,code=0)
> {noformat}
> Stack trace from sqlline.log:
> {noformat}
> 2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 27b54af4-b41f-0682-e50d-626de4eff68e: select * from
> dfs.`/tmp/tooManyFields.json` limit 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
> localhost. Skipping affinity to that host.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1
> using 1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1
> using 1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs,
> Average start: 6.956000 μs .
> 2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State to report: RUNNING
> 2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN
> o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available -
> probably you're running Drill with a JRE and not a JDK
> 2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested RUNNING -->
> FAILED
> 2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested FAILED -->
> FINISHED
> 2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException:
> Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0"
> grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> JaninoRuntimeException: Code attribute in class
> "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB
> Fragment 0:0
> [Error Id:
[jira] [Commented] (DRILL-5117) Compile error when query a json file with 1000+columns
[
https://issues.apache.org/jira/browse/DRILL-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769664#comment-15769664
]
Khurram Faraaz commented on DRILL-5117:
---
Verified on Drill 1.9.0 git commit ID : 4c1b420b
JSON file used to verify fix had 4095 keys in it. Data file is attached here
for future reference.
{noformat}
select * from `manyColsInJson.json` limit 1;
...
// No error/Exception was returned, instead correct result was returned.
{noformat}
> Compile error when query a json file with 1000+columns
> --
>
> Key: DRILL-5117
> URL: https://issues.apache.org/jira/browse/DRILL-5117
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
>
> Query failed with compile error when we querying a json file with
> 1000+columns:
> {noformat}
> 0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
> Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class
> "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
> (state=,code=0)
> {noformat}
> Stack trace from sqlline.log:
> {noformat}
> 2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 27b54af4-b41f-0682-e50d-626de4eff68e: select * from
> dfs.`/tmp/tooManyFields.json` limit 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
> localhost. Skipping affinity to that host.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1
> using 1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1
> using 1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs,
> Average start: 6.956000 μs .
> 2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State to report: RUNNING
> 2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN
> o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available -
> probably you're running Drill with a JRE and not a JDK
> 2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested RUNNING -->
> FAILED
> 2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested FAILED -->
> FINISHED
> 2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException:
> Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0"
> grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2
[jira] [Commented] (DRILL-5132) Context based dynamic parameterization of views
[
https://issues.apache.org/jira/browse/DRILL-5132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769661#comment-15769661
]
Nagarajan Chinnasamy commented on DRILL-5132:
-
Yes. Sounds good.
More questions :)
1. Is there a way for UDFs to store/set the auth-token (session_id) of the
external system somewhere?
- To re-use the auth-token for consecutive calls
- To update during session renewal
2. Demand drill for a re-authentication using custom authenticator if session
expired?
> Context based dynamic parameterization of views
> ---
>
> Key: DRILL-5132
> URL: https://issues.apache.org/jira/browse/DRILL-5132
> Project: Apache Drill
> Issue Type: Wish
> Components: Server
>Reporter: Nagarajan Chinnasamy
>Priority: Critical
> Labels: authentication, context, isolation, jdbcstorage,
> multi-tenancy, session-context, session-parameter
>
> *Requirement*
> Its known that Views in SQL cannot have custom dynamic parameters/variables.
> Please refer to [Justin
> Swanhart|http://stackoverflow.com/users/679236/justin-swanhart]'s response to
> [this SO
> question|http://stackoverflow.com/questions/2281890/can-i-create-view-with-parameter-in-mysql]
> in handling dynamic parameterization of views.
> [The PR #685|https://github.com/apache/drill/pull/685]
> [DRILL-5043|https://issues.apache.org/jira/browse/DRILL-5043?filter=-2]
> originated based on this requirement so that we could build views that can
> dynamically filter records based on some dynamic values (like current
> tenant-id, user role etc.)
> *Since Drill's basic unit is a View... having such built-in support can bring
> in dynamism into the whole game.*
> This feature can be utilized for:
> * *Data Isolation in Shared Multi-Tenant environments* based on Custom Tenant
> Discriminator Column
> * *Data Protection in building Chained Views* with Custom Dynamic Filters
> To explain this further, If we assume that:
> # As and when the user connection is established, we populate session context
> with session parameters such as:
> #* Tenant ID of the currently logged in user
> #* Roles of the currently logged in user
> # We expose the session context information through context-based-functions
> such as:
> #* *session_id* -- that returns unique id of the session
> #* *session_parameter('')* - that returns the value of the
> session parameter
> then a view created with the following kind of query:
> {code}
> create or replace view dynamic_filter_view as select
>a.field as a_field
>b.field as b_field
> from
>a_table as a
> left join
>b_table as b
> on
>a.bId = b.Id
> where
>session_parameter('tenantId')=a.tenantId
> {code}
> becomes a query that has built-in support for dynamic parameterization that
> only returns records of the tenant of the currently logged in user. This is a
> very useful feature in a shared-multi-tenant environment where data is
> isolated using multi-tenant-descriminator column 'tenantId'.
> When building chained views this feature will be useful in filtering records
> based on context based parameters.
> This feature will particularly be useful for data isolation / data protection
> with *jdbc storage plugins* where drill-authenticated-credentials are not
> passed to jdbc connection authentication. A jdbc storage has hard-coded,
> shared credentials. Hence the the responsibility of data isolation / data
> protection lies with Views themselves. Hence, the need for built-in support
> of context based dynamic parameters in Views.
> *Design/Implementation Considerations:*
> * Session parameters can be obtained through authenticators so that custom
> authenticators can return a HashMap of parameters obtained from external
> systems.
> * Introduce SessionContext to hold sessionId and sessionParameters
> * Implement context based functions session_id and session_parameter()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-5153) RESOURCE ERROR: Waited for 15000ms, but tasks for 'Get block maps' are not complete
[
https://issues.apache.org/jira/browse/DRILL-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Challapalli updated DRILL-5153:
-
Attachment: tera.log
Exception from the logs
{code}
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Waited for
15000ms, but tasks for 'Get block maps' are not complete. Total runnable size
2, parallelism 2.
[Error Id: 580e6c04-7096-4c09-9c7a-63e70c71d574 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:153)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.schedule.BlockMapBuilder.generateFileWork(BlockMapBuilder.java:75)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.dfs.easy.EasyGroupScan.initFromSelection(EasyGroupScan.java:133)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.dfs.easy.EasyGroupScan.(EasyGroupScan.java:104)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.text.TextFormatPlugin.getGroupScan(TextFormatPlugin.java:99)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:145)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:103)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:85)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:89)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:69)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:62)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:37)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.calcite.plan.vo
{code}
> RESOURCE ERROR: Waited for 15000ms, but tasks for 'Get block maps' are not
> complete
>
>
> Key: DRILL-5153
> URL: https://issues.apache.org/jira/browse/DRILL-5153
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC, Query Planning & Optimization
>Reporter: Rahul Challapalli
> Attachments: tera.log
>
>
> git.commit.id.abbrev=cf2b7c7
> The below query consistently fails on my 2 node cluster. I used the data set
> from the terasort benchmark
> {code}
> select * from dfs.`/drill/testdata/resource-manager/terasort-data` limit 1;
> Error: RESOURCE ERROR: Waited for 15000ms, but tasks for 'Get block maps' are
> not complete. Total runnable size 2, parallelism 2.
> [Error Id: 580e6c04-7096-4c09-9c7a-63e70c71d574 on qa-node182.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5153) RESOURCE ERROR: Waited for 15000ms, but tasks for 'Get block maps' are not complete
Rahul Challapalli created DRILL-5153:
Summary: RESOURCE ERROR: Waited for 15000ms, but tasks for 'Get
block maps' are not complete
Key: DRILL-5153
URL: https://issues.apache.org/jira/browse/DRILL-5153
Project: Apache Drill
Issue Type: Bug
Components: Execution - RPC, Query Planning & Optimization
Reporter: Rahul Challapalli
git.commit.id.abbrev=cf2b7c7
The below query consistently fails on my 2 node cluster. I used the data set
from the terasort benchmark
{code}
select * from dfs.`/drill/testdata/resource-manager/terasort-data` limit 1;
Error: RESOURCE ERROR: Waited for 15000ms, but tasks for 'Get block maps' are
not complete. Total runnable size 2, parallelism 2.
[Error Id: 580e6c04-7096-4c09-9c7a-63e70c71d574 on qa-node182.qa.lab:31010]
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
