[jira] [Created] (DRILL-5028) Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS.

[Hongze Zhang (JIRA)]

Hongze Zhang created DRILL-5028:
---

 Summary: Opening profiles page from web ui gets very slow when a 
lot of history files have been stored in HDFS or Local FS.
 Key: DRILL-5028
 URL: https://issues.apache.org/jira/browse/DRILL-5028
 Project: Apache Drill
  Issue Type: Improvement
  Components: Functions - Drill
Affects Versions: 1.8.0
Reporter: Hongze Zhang
 Fix For: Future


We have a Drill cluster with 20+ Nodes and we store all history profiles in 
hdfs. Without doing periodically cleans for hdfs, the profiles page gets slower 
while serving more queries.

Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching 
the latest 100 history profiles by default, I guess this operation blocks that 
page (Millions small files can be stored in the basePath), maybe we can try 
some other ways to reach the same goal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5027) ExternalSortBatch is inefficient, leaks files for large queries

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-5027:
--

Assignee: Paul Rogers

> ExternalSortBatch is inefficient, leaks files for large queries
> ---
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). Large queries need multiple 
> sort "phases" in which previously spilled runs are read back into memory, 
> merged, and again spilled. It is here that ESB has an issue. This process 
> correctly limit the amount of memory used, but at the cost or rewriting the 
> same data over and over.
> Consider current Drill behavior:
> {code}
> a b c d (re-spill)
> abcd e f g h (re-spill)
> abcefgh i j k
> {code}
> That is batches, a, b, c and d are re-spilled to create the combined abcd, 
> and so on. The same data is rewritten over and over.
> Note that spilled batches take no (direct) memory in Drill, and require only 
> a small on-heap memento. So, maintaining data on disk s "free". So, better 
> would be to re-spill only newer data:
> {code}
> a b c d (re-spill)
> abcd | e f g h (re-spill)
> abcd efgh | i j k
> {code}
> Where the bar indicates a moving point at which we've already merged and do 
> not need to do so again. If each letter is one unit of disk I/O, the original 
> method uses 35 units while the revised method uses 27 units.
> At some point the process may have to repeat by merging the second-generation 
> spill files and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (DRILL-5027) ExternalSortBatch is inefficient, leaks files for large queries

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5027:
---
Comment: was deleted

(was: Inspection reveals that the intermediate spill events do not clean up the 
previous generation of spill files until the Drillbit exits.  That is, if we 
spill files (a, b, c, d) to produce file e, the first four files remain on 
disk. In particular the {{mergeAndSpill}} method does not call 
{{BatchGroup.close()}} for each file that is merged into a new spilled batch.

However, the batches are removed from the {{spilledBatchGroups}} list, so that 
the ESB cannot remove them when the query completes.

The files are marked "delete on exit", so they are removed when the Drillbit 
exits.)

> ExternalSortBatch is inefficient, leaks files for large queries
> ---
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). Large queries need multiple 
> sort "phases" in which previously spilled runs are read back into memory, 
> merged, and again spilled. It is here that ESB has an issue. This process 
> correctly limit the amount of memory used, but at the cost or rewriting the 
> same data over and over.
> Consider current Drill behavior:
> {code}
> a b c d (re-spill)
> abcd e f g h (re-spill)
> abcefgh i j k
> {code}
> That is batches, a, b, c and d are re-spilled to create the combined abcd, 
> and so on. The same data is rewritten over and over.
> Note that spilled batches take no (direct) memory in Drill, and require only 
> a small on-heap memento. So, maintaining data on disk s "free". So, better 
> would be to re-spill only newer data:
> {code}
> a b c d (re-spill)
> abcd | e f g h (re-spill)
> abcd efgh | i j k
> {code}
> Where the bar indicates a moving point at which we've already merged and do 
> not need to do so again. If each letter is one unit of disk I/O, the original 
> method uses 35 units while the revised method uses 27 units.
> At some point the process may have to repeat by merging the second-generation 
> spill files and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (DRILL-5027) ExternalSortBatch is inefficient, leaks files for large queries

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5027:
---
Comment: was deleted

(was: Appears that the above comment is slightly off. Actual behavior is that 
when the number of previously-spilled batches hits some threshold, all the 
previously spilled batches are merged into a new batch.

This process does limit the amount of memory used, but at the cost or rewriting 
the same data over and over. Consider:
{code}
1 2 3 4 (respill)
1234 5 6 7 (respill)
1234567 8 9 10
{code}

That is batches, 1, 2, 3 and 4 are re-spilled to create 1234, and so on.

The same lead data is rewritten over and over. Better would be to re-spill only 
newer data:
{code}
1 2 3 4 (re-spill)
1234 | 5 6 7 9 (re-spill)
1234 5678 | 9 10 11
{code}

Where the bar indicates a moving point at which we've already merged and do not 
need to do so again.

At some point the process may have to repeat by merging the second-generation 
spill files and so on.)

> ExternalSortBatch is inefficient, leaks files for large queries
> ---
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). Large queries need multiple 
> sort "phases" in which previously spilled runs are read back into memory, 
> merged, and again spilled. It is here that ESB has an issue. This process 
> correctly limit the amount of memory used, but at the cost or rewriting the 
> same data over and over.
> Consider current Drill behavior:
> {code}
> a b c d (re-spill)
> abcd e f g h (re-spill)
> abcefgh i j k
> {code}
> That is batches, a, b, c and d are re-spilled to create the combined abcd, 
> and so on. The same data is rewritten over and over.
> Note that spilled batches take no (direct) memory in Drill, and require only 
> a small on-heap memento. So, maintaining data on disk s "free". So, better 
> would be to re-spill only newer data:
> {code}
> a b c d (re-spill)
> abcd | e f g h (re-spill)
> abcd efgh | i j k
> {code}
> Where the bar indicates a moving point at which we've already merged and do 
> not need to do so again. If each letter is one unit of disk I/O, the original 
> method uses 35 units while the revised method uses 27 units.
> At some point the process may have to repeat by merging the second-generation 
> spill files and so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5027) ExternalSortBatch is inefficient, leaks files for large queries

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5027:
---
Description: 
The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
needed to operate within a memory budget.

The sort happens in two phases:

1. Gather the incoming batches from the upstream operator, sort them, and spill 
to disk as needed.
2. Merge the "runs" spilled in step 1.

In most cases, the second step should run within the memory available for the 
first step (which is why severity is only Minor). Large queries need multiple 
sort "phases" in which previously spilled runs are read back into memory, 
merged, and again spilled. It is here that ESB has an issue. This process 
correctly limit the amount of memory used, but at the cost or rewriting the 
same data over and over.

Consider current Drill behavior:

{code}
a b c d (re-spill)
abcd e f g h (re-spill)
abcefgh i j k
{code}

That is batches, a, b, c and d are re-spilled to create the combined abcd, and 
so on. The same data is rewritten over and over.

Note that spilled batches take no (direct) memory in Drill, and require only a 
small on-heap memento. So, maintaining data on disk s "free". So, better would 
be to re-spill only newer data:

{code}
a b c d (re-spill)
abcd | e f g h (re-spill)
abcd efgh | i j k
{code}

Where the bar indicates a moving point at which we've already merged and do not 
need to do so again. If each letter is one unit of disk I/O, the original 
method uses 35 units while the revised method uses 27 units.

At some point the process may have to repeat by merging the second-generation 
spill files and so on.


  was:
The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
needed to operate within a memory budget.

The sort happens in two phases:

1. Gather the incoming batches from the upstream operator, sort them, and spill 
to disk as needed.
2. Merge the "runs" spilled in step 1.

In most cases, the second step should run within the memory available for the 
first step (which is why severity is only Minor). However, if the query is 
exceptionally large, then the second step can use excessive memory.

Here is why.

* In step 1, we create a series of n spill files. Each file contains batches of 
some maximum size, say 20 MB.
* In step 2, we must simultaneously open all the spilled files and read the 
first batch into memory in preparation for merging.

Suppose that the query has 1 TB of data. Suppose that 10 GB of memory is 
available. The result will be 1 TB / 1 GB = 1000 spill files. Suppose each 
batch in each file is 20 MB. A single-pass merge will need 20 MB * 1000 = 20 GB 
to operate. But, we only have 10 GB.

The typical solution is to perform the merge in multiple phases, with each 
phase reading and respilling only enough runs that will fit in memory. In the 
above case, the first phase would make two iterations, merging 500 GB in each 
using 10 GB of memory. The second phase would make a single iteration to merge 
the two first phase files.

The result is much disk I/O, and a requirement for sufficient disk spaces to 
store two sets of spill files (one from phase i-1, another for phase i). But, 
the query will complete within the memory budgeted.


> ExternalSortBatch is inefficient, leaks files for large queries
> ---
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). Large queries need multiple 
> sort "phases" in which previously spilled runs are read back into memory, 
> merged, and again spilled. It is here that ESB has an issue. This process 
> correctly limit the amount of memory used, but at the cost or rewriting the 
> same data over and over.
> Consider current Drill behavior:
> {code}
> a b c d (re-spill)
> abcd e f g h (re-spill)
> abcefgh i j k
> {code}
> That is batches, a, b, c and d are re-spilled to create the combined abcd, 
> and so on. The same data is rewritten over and over.
> Note that spilled batches take no (direct) memory in Drill, and require only 
> a small on-heap memento. So, maintaining data on disk s "free". So, better 
> would be to re-spill only newer data:
> {code}
> a b c d (re-spill)
> abcd | e f g h (re-spill)
> abcd efgh | i j k
> 

[jira] [Updated] (DRILL-5027) ExternalSortBatch is inefficient, leaks files for large queries

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5027:
---
Summary: ExternalSortBatch is inefficient, leaks files for large queries  
(was: ExternalSortBatch is efficient, leaks files for large queries)

> ExternalSortBatch is inefficient, leaks files for large queries
> ---
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). However, if the query is 
> exceptionally large, then the second step can use excessive memory.
> Here is why.
> * In step 1, we create a series of n spill files. Each file contains batches 
> of some maximum size, say 20 MB.
> * In step 2, we must simultaneously open all the spilled files and read the 
> first batch into memory in preparation for merging.
> Suppose that the query has 1 TB of data. Suppose that 10 GB of memory is 
> available. The result will be 1 TB / 1 GB = 1000 spill files. Suppose each 
> batch in each file is 20 MB. A single-pass merge will need 20 MB * 1000 = 20 
> GB to operate. But, we only have 10 GB.
> The typical solution is to perform the merge in multiple phases, with each 
> phase reading and respilling only enough runs that will fit in memory. In the 
> above case, the first phase would make two iterations, merging 500 GB in each 
> using 10 GB of memory. The second phase would make a single iteration to 
> merge the two first phase files.
> The result is much disk I/O, and a requirement for sufficient disk spaces to 
> store two sets of spill files (one from phase i-1, another for phase i). But, 
> the query will complete within the memory budgeted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5027) ExternalSortBatch is efficient, leaks files for large queries

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5027:
---
Summary: ExternalSortBatch is efficient, leaks files for large queries  
(was: ExternalSortBatch can use excessive memory for very large queries)

> ExternalSortBatch is efficient, leaks files for large queries
> -
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). However, if the query is 
> exceptionally large, then the second step can use excessive memory.
> Here is why.
> * In step 1, we create a series of n spill files. Each file contains batches 
> of some maximum size, say 20 MB.
> * In step 2, we must simultaneously open all the spilled files and read the 
> first batch into memory in preparation for merging.
> Suppose that the query has 1 TB of data. Suppose that 10 GB of memory is 
> available. The result will be 1 TB / 1 GB = 1000 spill files. Suppose each 
> batch in each file is 20 MB. A single-pass merge will need 20 MB * 1000 = 20 
> GB to operate. But, we only have 10 GB.
> The typical solution is to perform the merge in multiple phases, with each 
> phase reading and respilling only enough runs that will fit in memory. In the 
> above case, the first phase would make two iterations, merging 500 GB in each 
> using 10 GB of memory. The second phase would make a single iteration to 
> merge the two first phase files.
> The result is much disk I/O, and a requirement for sufficient disk spaces to 
> store two sets of spill files (one from phase i-1, another for phase i). But, 
> the query will complete within the memory budgeted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5027) ExternalSortBatch can use excessive memory for very large queries

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653085#comment-15653085
 ] 

Paul Rogers commented on DRILL-5027:


Inspection reveals that the intermediate spill events do not clean up the 
previous generation of spill files until the Drillbit exits.  That is, if we 
spill files (a, b, c, d) to produce file e, the first four files remain on 
disk. In particular the {{mergeAndSpill}} method does not call 
{{BatchGroup.close()}} for each file that is merged into a new spilled batch.

However, the batches are removed from the {{spilledBatchGroups}} list, so that 
the ESB cannot remove them when the query completes.

The files are marked "delete on exit", so they are removed when the Drillbit 
exits.

> ExternalSortBatch can use excessive memory for very large queries
> -
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). However, if the query is 
> exceptionally large, then the second step can use excessive memory.
> Here is why.
> * In step 1, we create a series of n spill files. Each file contains batches 
> of some maximum size, say 20 MB.
> * In step 2, we must simultaneously open all the spilled files and read the 
> first batch into memory in preparation for merging.
> Suppose that the query has 1 TB of data. Suppose that 10 GB of memory is 
> available. The result will be 1 TB / 1 GB = 1000 spill files. Suppose each 
> batch in each file is 20 MB. A single-pass merge will need 20 MB * 1000 = 20 
> GB to operate. But, we only have 10 GB.
> The typical solution is to perform the merge in multiple phases, with each 
> phase reading and respilling only enough runs that will fit in memory. In the 
> above case, the first phase would make two iterations, merging 500 GB in each 
> using 10 GB of memory. The second phase would make a single iteration to 
> merge the two first phase files.
> The result is much disk I/O, and a requirement for sufficient disk spaces to 
> store two sets of spill files (one from phase i-1, another for phase i). But, 
> the query will complete within the memory budgeted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5027) ExternalSortBatch can use excessive memory for very large queries

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653086#comment-15653086
 ] 

Paul Rogers commented on DRILL-5027:


Inspection reveals that the intermediate spill events do not clean up the 
previous generation of spill files until the Drillbit exits.  That is, if we 
spill files (a, b, c, d) to produce file e, the first four files remain on 
disk. In particular the {{mergeAndSpill}} method does not call 
{{BatchGroup.close()}} for each file that is merged into a new spilled batch.

However, the batches are removed from the {{spilledBatchGroups}} list, so that 
the ESB cannot remove them when the query completes.

The files are marked "delete on exit", so they are removed when the Drillbit 
exits.

> ExternalSortBatch can use excessive memory for very large queries
> -
>
> Key: DRILL-5027
> URL: https://issues.apache.org/jira/browse/DRILL-5027
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
> needed to operate within a memory budget.
> The sort happens in two phases:
> 1. Gather the incoming batches from the upstream operator, sort them, and 
> spill to disk as needed.
> 2. Merge the "runs" spilled in step 1.
> In most cases, the second step should run within the memory available for the 
> first step (which is why severity is only Minor). However, if the query is 
> exceptionally large, then the second step can use excessive memory.
> Here is why.
> * In step 1, we create a series of n spill files. Each file contains batches 
> of some maximum size, say 20 MB.
> * In step 2, we must simultaneously open all the spilled files and read the 
> first batch into memory in preparation for merging.
> Suppose that the query has 1 TB of data. Suppose that 10 GB of memory is 
> available. The result will be 1 TB / 1 GB = 1000 spill files. Suppose each 
> batch in each file is 20 MB. A single-pass merge will need 20 MB * 1000 = 20 
> GB to operate. But, we only have 10 GB.
> The typical solution is to perform the merge in multiple phases, with each 
> phase reading and respilling only enough runs that will fit in memory. In the 
> above case, the first phase would make two iterations, merging 500 GB in each 
> using 10 GB of memory. The second phase would make a single iteration to 
> merge the two first phase files.
> The result is much disk I/O, and a requirement for sufficient disk spaces to 
> store two sets of spill files (one from phase i-1, another for phase i). But, 
> the query will complete within the memory budgeted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5027) ExternalSortBatch can use excessive memory for very large queries

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5027:
--

 Summary: ExternalSortBatch can use excessive memory for very large 
queries
 Key: DRILL-5027
 URL: https://issues.apache.org/jira/browse/DRILL-5027
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor


The {{ExternalSortBatch}} (ESB) operator sorts data while spilling to disk as 
needed to operate within a memory budget.

The sort happens in two phases:

1. Gather the incoming batches from the upstream operator, sort them, and spill 
to disk as needed.
2. Merge the "runs" spilled in step 1.

In most cases, the second step should run within the memory available for the 
first step (which is why severity is only Minor). However, if the query is 
exceptionally large, then the second step can use excessive memory.

Here is why.

* In step 1, we create a series of n spill files. Each file contains batches of 
some maximum size, say 20 MB.
* In step 2, we must simultaneously open all the spilled files and read the 
first batch into memory in preparation for merging.

Suppose that the query has 1 TB of data. Suppose that 10 GB of memory is 
available. The result will be 1 TB / 1 GB = 1000 spill files. Suppose each 
batch in each file is 20 MB. A single-pass merge will need 20 MB * 1000 = 20 GB 
to operate. But, we only have 10 GB.

The typical solution is to perform the merge in multiple phases, with each 
phase reading and respilling only enough runs that will fit in memory. In the 
above case, the first phase would make two iterations, merging 500 GB in each 
using 10 GB of memory. The second phase would make a single iteration to merge 
the two first phase files.

The result is much disk I/O, and a requirement for sufficient disk spaces to 
store two sets of spill files (one from phase i-1, another for phase i). But, 
the query will complete within the memory budgeted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4706) Fragment planning causes Drillbits to read remote chunks when local copies are available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652712#comment-15652712
 ] 

ASF GitHub Bot commented on DRILL-4706:
---

Github user ppadma commented on the issue:

https://github.com/apache/drill/pull/639
  
Merged with latest code. All review comments taken care of. All tests pass 
with the option `store.parquet.use_local_affinity` = true and false, both.


> Fragment planning causes Drillbits to read remote chunks when local copies 
> are available
> 
>
> Key: DRILL-4706
> URL: https://issues.apache.org/jira/browse/DRILL-4706
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.6.0
> Environment: CentOS, RHEL
>Reporter: Kunal Khatua
>Assignee: Padma Penumarthy
>  Labels: performance, planning
>
> When a table (datasize=70GB) of 160 parquet files (each having a single 
> rowgroup and fitting within one chunk) is available on a 10-node setup with 
> replication=3 ; a pure data scan query causes about 2% of the data to be read 
> remotely. 
> Even with the creation of metadata cache, the planner is selecting a 
> sub-optimal plan of executing the SCAN fragments such that some of the data 
> is served from a remote server. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5026) ExternalSortBatch uses two memory allocators; one will do

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5026:
--

 Summary: ExternalSortBatch uses two memory allocators; one will do
 Key: DRILL-5026
 URL: https://issues.apache.org/jira/browse/DRILL-5026
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor


The {{ExternalSortBatch}} (ESB) operator performs a sort while spilling to disk 
to stay within a defined memory budget.

ESB uses two memory allocators:

* The operator allocator for most operations,
* The copier allocator (child of the operator allocator) for merge operations.

The copier allocator is used only when merging prior to spill.

However, since the ESB must manage within a single memory budget, the code is 
made more complex by using two allocators, and keeping track of the memory used 
in both.

Better to use a single allocator. When checking for the need to spill, the code 
already checks if enough memory is available for an in-memory sort, if needed. 
Similarly, the code should also check if sufficient memory is available for the 
copy step when needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5026) ExternalSortBatch uses two memory allocators; one will do

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-5026:
--

Assignee: Paul Rogers

> ExternalSortBatch uses two memory allocators; one will do
> -
>
> Key: DRILL-5026
> URL: https://issues.apache.org/jira/browse/DRILL-5026
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator performs a sort while spilling to 
> disk to stay within a defined memory budget.
> ESB uses two memory allocators:
> * The operator allocator for most operations,
> * The copier allocator (child of the operator allocator) for merge operations.
> The copier allocator is used only when merging prior to spill.
> However, since the ESB must manage within a single memory budget, the code is 
> made more complex by using two allocators, and keeping track of the memory 
> used in both.
> Better to use a single allocator. When checking for the need to spill, the 
> code already checks if enough memory is available for an in-memory sort, if 
> needed. Similarly, the code should also check if sufficient memory is 
> available for the copy step when needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652467#comment-15652467
 ] 

ASF GitHub Bot commented on DRILL-4935:
---

Github user zfong commented on the issue:

https://github.com/apache/drill/pull/647
  
@harrisonmebane - for pull requests submitted by non-committers, each week, 
one of the committers will do a batch commit of changes that have gone through 
review.  This is normally done towards the end of the week.  There may be an 
exception this week because as @paul-rogers has noted, we're in the middle of 
trying to push out a release candidate build for the 1.9 release.


> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
> Fix For: Future
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652415#comment-15652415
 ] 

ASF GitHub Bot commented on DRILL-4935:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/647
  
The committers are in the middle of doing a release. Once that is done, you 
need one of them to do a quick review and merge your pull request.


> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
> Fix For: Future
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652341#comment-15652341
 ] 

ASF GitHub Bot commented on DRILL-4935:
---

Github user harrisonmebane commented on the issue:

https://github.com/apache/drill/pull/647
  
Do I need to do anything more to push this forward?


> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
> Fix For: Future
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5025) ExternalSortBatch provides weak control over spill file size

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5025:
--

 Summary: ExternalSortBatch provides weak control over spill file 
size
 Key: DRILL-5025
 URL: https://issues.apache.org/jira/browse/DRILL-5025
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor


The ExternalSortBatch (ESB) operator sorts records while spilling to disk to 
control memory use. The size of the spill file is not easy to control. It is a 
function of the accumulated batches size (half of the accumulated total), which 
is determined by either the memory budget or the 
{{drill.exec.sort.external.group.size}} parameter. (But, even with the 
parameter, the actual file size is still half the accumulated batches.)

The proposed solution is to provide an explicit parameter that sets the maximum 
spill file size: {{drill.exec.sort.external.spill.size}}. If the ESB needs to 
spill more than this amount of data, ESB should split the spill into multiple 
files.

The spill.size should be in bytes (or MB). (A size in records makes the file 
size data-dependent, which would not be helpful.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4373) Drill and Hive have incompatible timestamp representations in parquet

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-4373:

Reviewer: Krystal  (was: Rahul Challapalli)

[~knguyen] Can you look at this and verify the fix? [~rkins], since you don't 
have the bandwidth, please guide Krystal on how to go about testing and 
verifying this. 

> Drill and Hive have incompatible timestamp representations in parquet
> -
>
> Key: DRILL-4373
> URL: https://issues.apache.org/jira/browse/DRILL-4373
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Hive, Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>  Labels: doc-impacting
> Fix For: 1.9.0
>
>
> git.commit.id.abbrev=83d460c
> I created a parquet file with a timestamp type using Drill. Now if I define a 
> hive table on top of the parquet file and use "timestamp" as the column type, 
> drill fails to read the hive table through the hive storage plugin
> Implementation: 
> Added int96 to timestamp converter for both parquet readers and controling it 
> by system / session option "store.parquet.int96_as_timestamp".
> The value of the option is false by default for the proper work of the old 
> query scripts with the "convert_from TIMESTAMP_IMPALA" function.
> When the option is true using of that function is unnesessary and can lead to 
> the query fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652095#comment-15652095
 ] 

ASF GitHub Bot commented on DRILL-4935:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/647
  
Great! LGTM.


> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
> Fix For: Future
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4979) Make dataport configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652032#comment-15652032
 ] 

ASF GitHub Bot commented on DRILL-4979:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/649
  
Thanks for the explanation.
LGTM


> Make dataport configurable
> --
>
> Key: DRILL-4979
> URL: https://issues.apache.org/jira/browse/DRILL-4979
> Project: Apache Drill
>  Issue Type: New Feature
>  Components:  Server
>Affects Versions: 1.8.0
> Environment: Scheduling drillbits with Apache Mesos+Aurora
>Reporter: Uwe L. Korn
>
> Currently the dataport of a Drillbit is fixed to +1 on the control port. In a 
> dynamic execution environment like Apache Mesos+Aurora, each port is 
> allocated by the scheduler and then passed on to the application process. 
> There is no possibility or guarantee to allocate two consecutive ports. 
> Therefore, to run Drill in this environment, the dataport of the drillbit 
> also needs to configurable by the scheduler. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4979) Make dataport configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651993#comment-15651993
 ] 

ASF GitHub Bot commented on DRILL-4979:
---

Github user xhochy commented on the issue:

https://github.com/apache/drill/pull/649
  
No, this patch is actually functioning as the data port is already included 
in the probuf: 
https://github.com/apache/drill/blob/master/protocol/src/main/protobuf/Coordination.proto#L11
 The `+1 logic` is only used in the initialization code for the port.

Also I did a test run with 5 drillbits and random dataports assigned to 
them. They each were able to talk to them showing that the correct ports are 
communicated.


> Make dataport configurable
> --
>
> Key: DRILL-4979
> URL: https://issues.apache.org/jira/browse/DRILL-4979
> Project: Apache Drill
>  Issue Type: New Feature
>  Components:  Server
>Affects Versions: 1.8.0
> Environment: Scheduling drillbits with Apache Mesos+Aurora
>Reporter: Uwe L. Korn
>
> Currently the dataport of a Drillbit is fixed to +1 on the control port. In a 
> dynamic execution environment like Apache Mesos+Aurora, each port is 
> allocated by the scheduler and then passed on to the application process. 
> There is no possibility or guarantee to allocate two consecutive ports. 
> Therefore, to run Drill in this environment, the dataport of the drillbit 
> also needs to configurable by the scheduler. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651922#comment-15651922
 ] 

ASF GitHub Bot commented on DRILL-4935:
---

Github user harrisonmebane commented on the issue:

https://github.com/apache/drill/pull/647
  
I have tested this fix manually, in the following way:

* Deploy Drill in Docker containers on an existing cluster on AWS, with the 
line 
```export DRILL_HOST_NAME=`curl 
http://169.254.169.254/latest/meta-data/local-ipv4` ``` in `drill-env.sh`, to 
ensure that the variable is populated with the host machine's IP.
* Start a Drill session in the docker container and run `select * from 
sys.drillbits;`.  The result was: 
```
+++---++--+
|hostname| user_port  | control_port  | data_port  | current  |
+++---++--+
| 172.31.21.207  | 31010  | 31011 | 31012  | false|
| 172.31.29.130  | 31010  | 31011 | 31012  | true |
| 172.31.22.200  | 31010  | 31011 | 31012  | false|
+++---++--+
```
* On one of the drillbits, comment out the `DRILL_HOST_NAME` line in 
`drill-env.sh`, unset the variable, and restart the drillbit.
* Log into Drill shell again and run `select * from sys.drillbits;`  Result 
is now:
```
+++---++--+
|hostname| user_port  | control_port  | data_port  | current  |
+++---++--+
| 172.31.22.200  | 31010  | 31011 | 31012  | false|
| a53b37888f62   | 31010  | 31011 | 31012  | true |
| 172.31.29.130  | 31010  | 31011 | 31012  | false|
+++---++--+
```
* Running queries that require the other nodes results in a `Error: SYSTEM 
ERROR: UnresolvedAddressException`, indicating that the new address registered 
on Zookeeper no longer works




> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
> Fix For: Future
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5024) CTAS with LIMIT 0 query in SELECT stmt does not create parquet file

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5024:
-

 Summary: CTAS with LIMIT 0 query in SELECT stmt does not create 
parquet file
 Key: DRILL-5024
 URL: https://issues.apache.org/jira/browse/DRILL-5024
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.8.0
Reporter: Khurram Faraaz


Note that CTAS was successful
{noformat}
0: jdbc:drill:schema=dfs.tmp> create table regtbl_w0rows as select * from 
typeall_l LIMIT 0;
+---++
| Fragment  | Number of records written  |
+---++
| 0_0   | 0  |
+---++
1 row selected (0.51 seconds)
{noformat}

But a SELECT on CTAS created file fails.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from regtbl_w0rows;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 27: Table 
'regtbl_w0rows' not found

SQL Query null

[Error Id: 0569cf98-3800-43ee-b635-aa101b016d46 on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

DROP on the CTAS created table also fails
{noformat}
0: jdbc:drill:schema=dfs.tmp> drop table regtbl_w0rows;
Error: VALIDATION ERROR: Table [regtbl_w0rows] not found


[Error Id: fb0b1ea8-f76d-42e2-b69c-4beae2798bdf on centos-01.qa.lab:31010] 
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}

Verified that CTAS did not create a physical file in dfs.tmp schema
{noformat}
[test@cent01 bin]# hadoop fs -ls /tmp/regtbl_w0rows
ls: `/tmp/regtbl_w0rows': No such file or directory
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5007) Dynamic UDF lazy-init does not work correctly in multi-node cluster

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-5007:
---
Fix Version/s: 1.9.0

> Dynamic UDF lazy-init does not work correctly in multi-node cluster
> ---
>
> Key: DRILL-5007
> URL: https://issues.apache.org/jira/browse/DRILL-5007
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Paul Rogers
> Fix For: 1.9.0
>
>
> Reproduced in .csv file with 25 different fields, 2712474 lines and 570 MB 
> amount of data.
> When I registered jar in 1st node and ran long query with function for first 
> time I had got error:
> {quote}
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to 
> materialize incoming schema.  Errors:
>  
> Error in expression at index 0.  Error: Missing function 
> implementation:[f(VARCHAR-OPTIONAL, VARCHAR-OPTIONAL).  Full expression: 
> null..
> {quote}
> When I tried to run this query second time it finished correctly. It seems 
> another nodes did not get new function. 
> So lazy-init does not work well on another nodes before query fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5009) Query with a simple join fails on Hive generated parquet

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651488#comment-15651488
 ] 

ASF GitHub Bot commented on DRILL-5009:
---

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/651
  
+1


> Query with a simple join fails on Hive generated parquet
> 
>
> Key: DRILL-5009
> URL: https://issues.apache.org/jira/browse/DRILL-5009
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
> Environment: Commit ID: 5a439424594eb10d113163eaa1fdf8034f387235c
> 1.9.0 SNAPSHOT - Nov 5 2016
>Reporter: Abhishek Girish
>Assignee: Parth Chandra
>Priority: Blocker
> Fix For: 1.9.0
>
> Attachments: DRILL-5009.log.txt
>
>
> Query: 
> {code}
> SELECT *
> FROM store_sales ss, customer c
> WHERE  ss.ss_customer_sk = c.c_customer_sk 
> LIMIT 1; 
> {code}
> Error:
> {code}
> Error: SYSTEM ERROR: IOException: End of stream reached while initializing 
> buffered reader.
> Fragment 2:0
> [Error Id: 93726aea-1d62-4e7c-a2bf-1d7cc1e834e4 on abhi1:31010]
>   (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet 
> record reader.
> ...
> ...
>  Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Error 
> opening or reading metadata for parquet file at location: customer.parquet
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.():145
> 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.():59
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.():96
> 
> org.apache.drill.exec.store.parquet.columnreaders.NullableColumnReader.():39
> 
> org.apache.drill.exec.store.parquet.columnreaders.NullableFixedByteAlignedReaders$NullableFixedByteAlignedReader.():58
> 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory.getNullableColumnReader():252
> 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory.createFixedColumnReader():186
> 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.setup():402
> org.apache.drill.exec.physical.impl.ScanBatch.next():212
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.broadcastsender.BroadcastSenderRootExec.innerNext():95
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> ...
> {code}
> Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5020) ExternalSortBatch has inconsistent notions of the memory limit

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5020:
---
Summary: ExternalSortBatch has inconsistent notions of the memory limit  
(was: ExernalSortBatch has inconsistent notions of the memory limit)

> ExternalSortBatch has inconsistent notions of the memory limit
> --
>
> Key: DRILL-5020
> URL: https://issues.apache.org/jira/browse/DRILL-5020
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The memory limit cases for the spill-needed test seem inconsistent:
> For the test for in-memory sort:
> {code}
> long currentlyAvailable =  popConfig.getMaxAllocation() - 
> oAllocator.getAllocatedMemory();
> {code}
> For reaching the memory limit:
> {code}
> oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
> {code}
> That is, one uses {{oAllocator.getLimit}} ("the current maximum limit this 
> allocator imposes"), the other uses {{popConfig.getMaxAllocation}} ("The 
> maximum memory this operator can allocate".)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5020) ExernalSortBatch has inconsistent notions of the memory limit

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-5020:
--

Assignee: Paul Rogers

> ExernalSortBatch has inconsistent notions of the memory limit
> -
>
> Key: DRILL-5020
> URL: https://issues.apache.org/jira/browse/DRILL-5020
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The memory limit cases for the spill-needed test seem inconsistent:
> For the test for in-memory sort:
> {code}
> long currentlyAvailable =  popConfig.getMaxAllocation() - 
> oAllocator.getAllocatedMemory();
> {code}
> For reaching the memory limit:
> {code}
> oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
> {code}
> That is, one uses {{oAllocator.getLimit}} ("the current maximum limit this 
> allocator imposes"), the other uses {{popConfig.getMaxAllocation}} ("The 
> maximum memory this operator can allocate".)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5012) External Sort Batch does not check free disk space before writing

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5012:
---
Summary: External Sort Batch does not check free disk space before writing  
(was: External Sort Batch does not check free space before writing)

> External Sort Batch does not check free disk space before writing
> -
>
> Key: DRILL-5012
> URL: https://issues.apache.org/jira/browse/DRILL-5012
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> The ExternalSortBatch operator sorts data and spills to disk to control 
> memory usage. The ESB accepts a configured set of output directories.
> The ESB does not check if sufficient space is available in a directory before 
> writing to it; ESB requires that the admin ensure space is available.
> Desired fix: skip a directory if it does not contain space to hold the target 
> spill file. (That is, write to the other directories if that have space.)
> The fix is not fail-safe: two threads may both check and both write. However, 
> the fix reduces the chances of an out-of-disk error for the most obvious 
> cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5014) ExternalSortBatch cache size, spill count differs from config setting

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-5014:
--

Assignee: Paul Rogers

> ExternalSortBatch cache size, spill count differs from config setting
> -
>
> Key: DRILL-5014
> URL: https://issues.apache.org/jira/browse/DRILL-5014
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The ExternalSortBatch (ESB) operator sorts data while spilling to disk to 
> remain within a defined memory footprint. Spilling occurs based on a number 
> of factors. Among those are two config parameters:
> * {{drill.exec.sort.external.spill.group.size}}: The number of batches to 
> spill per spill event.
> * {{drill.exec.sort.external.spill.threshold}}: The number of batches to 
> accumulate in memory before starting a spill event.
> The expected behavior would be:
> * After the accumulated batches exceeds the threshold, and
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> That is if the threshold is 200, and the size is 150, we should accumulate 
> 200 batches, then spill 150 of them (leaving 50) and repeat.
> The actual behavior is:
> * When the accumulated records exceeds the threshold and,
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> The above can leave more batches in memory than desired, and spill more than 
> desired.
> The actual behavior for the (threshold=200, size=150) case is:
> {code}
> Before spilling, buffered batch count: 201
> After spilling, buffered batch count: 101
> Before spilling, buffered batch count: 251
> After spilling, buffered batch count: 126
> Before spilling, buffered batch count: 276
> After spilling, buffered batch count: 138
> Before spilling, buffered batch count: 288
> After spilling, buffered batch count: 144
> Before spilling, buffered batch count: 294
> After spilling, buffered batch count: 147
> Before spilling, buffered batch count: 297
> After spilling, buffered batch count: 149
> Before spilling, buffered batch count: 299
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> {code}
> In short, the actual number of batches retained in memory is twice the spill 
> size, **NOT** the number set in the threshold. As a result, the ESB operator 
> will use more memory than expected.
> The work-around is to set a batch size that is half the threshold so that the 
> batch size (used in spill decisions) matches the actual spill count (as 
> implemented by the code.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5022) ExternalSortBatch sets two different limits for "copier" memory

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-5022:
--

Assignee: Paul Rogers

> ExternalSortBatch sets two different limits for "copier" memory
> ---
>
> Key: DRILL-5022
> URL: https://issues.apache.org/jira/browse/DRILL-5022
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The {{ExternalSortBatch}} (ESB) operator sorts rows and supports spilling to 
> disk to operate within a set memory budget.
> A key step in disk-based sorting is to merge "runs" of previously-sorted 
> records. ESB does this with a class created from the 
> {{PriorityQueueCopierTemplate}}, called the "copier" in the code.
> The sort runs are represented by record batches, each with an indirection 
> vector (AKA {{SelectionVector}}) that point to the records in sort order.
> The copier restructures the incoming runs: copying from the original batches 
> (from positions given by the indirection vector) into new output vectors in 
> sorted order. To do this work, the copier must allocate new vectors to hold 
> the merged data. These vectors consume memory, and must fit into the overall 
> memory budget assigned to the ESB.
> As it turns out, the ESB code has two conflicting ways of setting the limit. 
> One is hard-coded:
> {code}
>   private static final int COPIER_BATCH_MEM_LIMIT = 256 * 1024;
> {code}
> The other comes from config parameters:
> {code}
>   public static final long INITIAL_ALLOCATION = 10_000_000;
>   public static final long MAX_ALLOCATION = 20_000_000;
> copierAllocator = oAllocator.newChildAllocator(oAllocator.getName() + 
> ":copier",
> PriorityQueueCopier.INITIAL_ALLOCATION, 
> PriorityQueueCopier.MAX_ALLOCATION);
> {code}
> Strangely, the config parameters are used to set aside memory for the copier 
> to use. But, the {{COPIER_BATCH_MEM_LIMIT}} is used to determine how large of 
> a merged batch to actually create.
> The result is that we set aside 10 MB of memory, but use only 256K of it, 
> wasting 9 MB.
> This ticket asks to:
> * Determine the proper merged batch size.
> * Use that limit to set the memory allocation for the copier.
> Elsewhere in Drill batch sizes tend to be on the order of 32K records. In the 
> ESB, the low {{COPIER_BATCH_MEM_LIMIT}} tends to favor smaller batches: A 
> test case has a row width of 114 bytes, and produces batches of just 2299 
> records. So, likely the proper choice is the larger 10 MB memory allocator 
> limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[Krystal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krystal closed DRILL-4870.
--

Verified on Mac the without setting JAVA_HOME, drill does set it and sqlline in 
embedded mode starts successfully.

> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Chunhui Shi
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5009) Query with a simple join fails on Hive generated parquet

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651444#comment-15651444
 ] 

ASF GitHub Bot commented on DRILL-5009:
---

GitHub user parthchandra opened a pull request:

https://github.com/apache/drill/pull/651

DRILL-5009: Skip reading of empty row groups while reading Parquet me…

…tadata.

We will no longer attempt to scan such row groups.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/parthchandra/drill DRILL-5009

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/651.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #651


commit 6a5a6f0576a2c83617a5a980d61ae82a49162235
Author: Parth Chandra 
Date:   2016-11-08T04:29:23Z

DRILL-5009: Skip reading of empty row groups while reading Parquet metadata.
We will no longer attempt to scan such row groups.




> Query with a simple join fails on Hive generated parquet
> 
>
> Key: DRILL-5009
> URL: https://issues.apache.org/jira/browse/DRILL-5009
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.9.0
> Environment: Commit ID: 5a439424594eb10d113163eaa1fdf8034f387235c
> 1.9.0 SNAPSHOT - Nov 5 2016
>Reporter: Abhishek Girish
>Assignee: Parth Chandra
>Priority: Blocker
> Fix For: 1.9.0
>
> Attachments: DRILL-5009.log.txt
>
>
> Query: 
> {code}
> SELECT *
> FROM store_sales ss, customer c
> WHERE  ss.ss_customer_sk = c.c_customer_sk 
> LIMIT 1; 
> {code}
> Error:
> {code}
> Error: SYSTEM ERROR: IOException: End of stream reached while initializing 
> buffered reader.
> Fragment 2:0
> [Error Id: 93726aea-1d62-4e7c-a2bf-1d7cc1e834e4 on abhi1:31010]
>   (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet 
> record reader.
> ...
> ...
>  Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Error 
> opening or reading metadata for parquet file at location: customer.parquet
> org.apache.drill.exec.store.parquet.columnreaders.PageReader.():145
> 
> org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.():59
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.():96
> 
> org.apache.drill.exec.store.parquet.columnreaders.NullableColumnReader.():39
> 
> org.apache.drill.exec.store.parquet.columnreaders.NullableFixedByteAlignedReaders$NullableFixedByteAlignedReader.():58
> 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory.getNullableColumnReader():252
> 
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory.createFixedColumnReader():186
> 
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.setup():402
> org.apache.drill.exec.physical.impl.ScanBatch.next():212
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.broadcastsender.BroadcastSenderRootExec.innerNext():95
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> ...
> {code}
> Log attached. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651436#comment-15651436
 ] 

ASF GitHub Bot commented on DRILL-4980:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/644#discussion_r87232002
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
 ---
@@ -59,19 +59,24 @@
*/
   public static final long JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH = 2440588;
   /**
-   * All old parquet files (which haven't "is.date.correct=true" property 
in metadata) have
-   * a corrupt date shift: {@value} days or 2 * {@value 
#JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH}
+   * All old parquet files (which haven't "is.date.correct=true" or 
"parquet-writer.version" properties
+   * in metadata) have a corrupt date shift: {@value} days or 2 * {@value 
#JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH}
*/
   public static final long CORRECT_CORRUPT_DATE_SHIFT = 2 * 
JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH;
-  // The year 5000 (or 1106685 day from Unix epoch) is chosen as the 
threshold for auto-detecting date corruption.
-  // This balances two possible cases of bad auto-correction. External 
tools writing dates in the future will not
-  // be shifted unless they are past this threshold (and we cannot 
identify them as external files based on the metadata).
-  // On the other hand, historical dates written with Drill wouldn't risk 
being incorrectly shifted unless they were
-  // something like 10,000 years in the past.
   private static final Chronology UTC = 
org.joda.time.chrono.ISOChronology.getInstanceUTC();
+  /**
+   * The year 5000 (or 1106685 day from Unix epoch) is chosen as the 
threshold for auto-detecting date corruption.
+   * This balances two possible cases of bad auto-correction. External 
tools writing dates in the future will not
+   * be shifted unless they are past this threshold (and we cannot 
identify them as external files based on the metadata).
+   * On the other hand, historical dates written with Drill wouldn't risk 
being incorrectly shifted unless they were
+   * something like 10,000 years in the past.
+   */
   public static final int DATE_CORRUPTION_THRESHOLD =
   (int) (UTC.getDateTimeMillis(5000, 1, 1, 0) / 
DateTimeConstants.MILLIS_PER_DAY);
-
+  /**
+   * The version of drill parquet writer with date values corruption fix
--- End diff --

Maybe explain this a bit better. Something like:

Version 2 (and later) of the Drill Parquet writer uses the date format 
described (in the Parquet spec? URL?). Prior versions had dates formatted (copy 
description from above.)


> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.9.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651437#comment-15651437
 ] 

ASF GitHub Bot commented on DRILL-4980:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/644#discussion_r87231425
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java 
---
@@ -185,7 +185,8 @@ private Metadata(FileSystem fs, ParquetFormatConfig 
formatConfig) {
 childFiles.add(file);
   }
 }
-ParquetTableMetadata_v3 parquetTableMetadata = new 
ParquetTableMetadata_v3(true);
+ParquetTableMetadata_v3 parquetTableMetadata = new 
ParquetTableMetadata_v3(DrillVersionInfo.getVersion(),
+ParquetWriter.WRITER_VERSION);
--- End diff --

I'm a bit confused. The writer version applies to the Parquet files which 
Drill writes. (Or, at least, that was the intention.)

Here, we're talking about metadata. There may well be a metadata writer, 
but that should be a different writer, with a different version.

Not sure we want to initialize the metadata object with the current writer 
version: there seems to be no correlation between the metadata object and the 
writer version.

On the other hand, the metadata can certainly hold the writer version, but 
it must be the value read from the Parquet file itself; not a value set by the 
code. Else, we have the difficult problem of making sure that the code-set 
version number agrees with the actual version number in the file.


> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.9.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651439#comment-15651439
 ] 

ASF GitHub Bot commented on DRILL-4980:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/644#discussion_r87232692
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
 ---
@@ -189,10 +194,14 @@ public static DateCorruptionStatus 
detectCorruptDates(ParquetMetadata footer,
 
 String createdBy = footer.getFileMetaData().getCreatedBy();
 String drillVersion = 
footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.DRILL_VERSION_PROPERTY);
-String isDateCorrect = 
footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.IS_DATE_CORRECT_PROPERTY);
+String writerVersion = 
footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.WRITER_VERSION_PROPERTY);
+// This flag can be present in parquet files which were generated with 
1.9.0-SNAPSHOT drill version
+final String isDateCorrectFlag = "is.date.correct";
--- End diff --

Maybe here you want to special case the "is.date.correct" flag.

1) If the writer version is present, use it.
2) If "is.date.correct" is present, set the writer version to "2".
3) If neither are present, set the writer version to 1.

That way we don't have to have (much) extra logic for the "is.date.correct" 
handling.


> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.9.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651438#comment-15651438
 ] 

ASF GitHub Bot commented on DRILL-4980:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/644#discussion_r87232227
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
 ---
@@ -59,19 +59,24 @@
*/
   public static final long JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH = 2440588;
   /**
-   * All old parquet files (which haven't "is.date.correct=true" property 
in metadata) have
-   * a corrupt date shift: {@value} days or 2 * {@value 
#JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH}
+   * All old parquet files (which haven't "is.date.correct=true" or 
"parquet-writer.version" properties
+   * in metadata) have a corrupt date shift: {@value} days or 2 * {@value 
#JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH}
*/
   public static final long CORRECT_CORRUPT_DATE_SHIFT = 2 * 
JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH;
-  // The year 5000 (or 1106685 day from Unix epoch) is chosen as the 
threshold for auto-detecting date corruption.
-  // This balances two possible cases of bad auto-correction. External 
tools writing dates in the future will not
-  // be shifted unless they are past this threshold (and we cannot 
identify them as external files based on the metadata).
-  // On the other hand, historical dates written with Drill wouldn't risk 
being incorrectly shifted unless they were
-  // something like 10,000 years in the past.
   private static final Chronology UTC = 
org.joda.time.chrono.ISOChronology.getInstanceUTC();
+  /**
+   * The year 5000 (or 1106685 day from Unix epoch) is chosen as the 
threshold for auto-detecting date corruption.
+   * This balances two possible cases of bad auto-correction. External 
tools writing dates in the future will not
+   * be shifted unless they are past this threshold (and we cannot 
identify them as external files based on the metadata).
+   * On the other hand, historical dates written with Drill wouldn't risk 
being incorrectly shifted unless they were
+   * something like 10,000 years in the past.
+   */
   public static final int DATE_CORRUPTION_THRESHOLD =
   (int) (UTC.getDateTimeMillis(5000, 1, 1, 0) / 
DateTimeConstants.MILLIS_PER_DAY);
-
+  /**
+   * The version of drill parquet writer with date values corruption fix
+   */
+  public static final int DRILL_WRITER_VERSION_WITHOUT_CORRUPTION = 2;
--- End diff --

Maybe call this DRILL_WRITER_VERSION_STD_DATE_FORMAT

The old format was not "corrupted", it just used a date format that was 
non-standard.


> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.9.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651435#comment-15651435
 ] 

ASF GitHub Bot commented on DRILL-4980:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/644#discussion_r87232983
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetWriter.java
 ---
@@ -40,6 +40,16 @@
 public class ParquetWriter extends AbstractWriter {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(ParquetWriter.class);
 
+/** Version of Drill's Parquet writer. Increment this version (by 1) any 
time we make any format change to the file.
+ * Format changes include
+ * 1) supporting new data types,
+ * 2) changes to the format of data fields,
+ * 3) adding new metadata to the file footer, etc.
+ * Newer readers must be able to read old files. The Writer version tells 
the Parquet reader how to interpret fields
--- End diff --

This is Javadoc, so needs HTML formatting. For the list:


Supporting new data types,
...



> Upgrading of the approach of parquet date correctness status detection
> --
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 1.8.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.9.0
>
>
> This jira is an addition for the 
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be 
> upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5007) Dynamic UDF lazy-init does not work correctly in multi-node cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651410#comment-15651410
 ] 

ASF GitHub Bot commented on DRILL-5007:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/650
  
+1. Good catch guys!


> Dynamic UDF lazy-init does not work correctly in multi-node cluster
> ---
>
> Key: DRILL-5007
> URL: https://issues.apache.org/jira/browse/DRILL-5007
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Paul Rogers
>
> Reproduced in .csv file with 25 different fields, 2712474 lines and 570 MB 
> amount of data.
> When I registered jar in 1st node and ran long query with function for first 
> time I had got error:
> {quote}
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to 
> materialize incoming schema.  Errors:
>  
> Error in expression at index 0.  Error: Missing function 
> implementation:[f(VARCHAR-OPTIONAL, VARCHAR-OPTIONAL).  Full expression: 
> null..
> {quote}
> When I tried to run this query second time it finished correctly. It seems 
> another nodes did not get new function. 
> So lazy-init does not work well on another nodes before query fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5007) Dynamic UDF lazy-init does not work correctly in multi-node cluster

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong reassigned DRILL-5007:
---

Assignee: Paul Rogers  (was: Arina Ielchiieva)

Assigning to [~Paul.Rogers] for review.

> Dynamic UDF lazy-init does not work correctly in multi-node cluster
> ---
>
> Key: DRILL-5007
> URL: https://issues.apache.org/jira/browse/DRILL-5007
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Paul Rogers
>
> Reproduced in .csv file with 25 different fields, 2712474 lines and 570 MB 
> amount of data.
> When I registered jar in 1st node and ran long query with function for first 
> time I had got error:
> {quote}
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to 
> materialize incoming schema.  Errors:
>  
> Error in expression at index 0.  Error: Missing function 
> implementation:[f(VARCHAR-OPTIONAL, VARCHAR-OPTIONAL).  Full expression: 
> null..
> {quote}
> When I tried to run this query second time it finished correctly. It seems 
> another nodes did not get new function. 
> So lazy-init does not work well on another nodes before query fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5010) Equality join condition is treated as a MergeJoin and not as a HashJoin.

[Zelaine Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651264#comment-15651264
 ] 

Zelaine Fong commented on DRILL-5010:
-

When you're projecting all columns, the cost of doing the sort for the 
sort-merge join is higher.

> Equality join condition is treated as a MergeJoin and not as a HashJoin.
> 
>
> Key: DRILL-5010
> URL: https://issues.apache.org/jira/browse/DRILL-5010
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>
> Equality join condition is treated as a MergeJoin and not as a HashJoin.
> Drill 1.9.0 git commit ID: 83513daf
> Projecting the join columns results in merge join, whereas it should be doing 
> a HashJoin.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select t1.intKey, t2.intKey from `left.json` 
> t1, `right_all_nulls.json` t2 
> WHERE t1.intKey = t2.intKey OR ( t1.intKey IS NULL AND t2.intKey IS NULL);
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(intKey=[$0], intKey0=[$1])
> 00-02Project(intKey=[$0], intKey0=[$1])
> 00-03  MergeJoin(condition=[IS NOT DISTINCT FROM($0, $1)], 
> joinType=[inner])
> 00-05SelectionVectorRemover
> 00-07  Sort(sort0=[$0], dir0=[ASC])
> 00-09Scan(groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/left.json, numFiles=1, columns=[`intKey`], 
> files=[maprfs:///tmp/left.json]]])
> 00-04Project(intKey0=[$0])
> 00-06  SelectionVectorRemover
> 00-08Sort(sort0=[$0], dir0=[ASC])
> 00-10  Scan(groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/right_all_nulls.json, numFiles=1, 
> columns=[`intKey`], files=[maprfs:///tmp/right_all_nulls.json]]])
> {noformat}
> Note that HashAgg and HashJoin were enabled.
> {noformat}
> | planner.enable_hashagg | BOOLEAN  | SYSTEM  | DEFAULT  
> | null| null| true  | null   |
> | planner.enable_hashjoin| BOOLEAN  | SYSTEM  | DEFAULT  
> | null| null| true  | null   |
> {noformat}
> Doing a SELECT  results in a HashJoin in the query plan
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select * from `left.json` t1, 
> `right_all_nulls.json` t2 
> WHERE t1.intKey = t2.intKey OR ( t1.intKey IS NULL AND t2.intKey IS NULL);
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  ProjectAllowDup(*=[$0], *0=[$1])
> 00-02Project(T46¦¦*=[$0], T47¦¦*=[$2])
> 00-03  HashJoin(condition=[IS NOT DISTINCT FROM($1, $3)], 
> joinType=[inner])
> 00-04Project(T47¦¦*=[$0], intKey0=[$1])
> 00-06  Project(T47¦¦*=[$0], intKey=[$1])
> 00-08Scan(groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/right_all_nulls.json, numFiles=1, columns=[`*`], 
> files=[maprfs:///tmp/right_all_nulls.json]]])
> 00-05Project(T46¦¦*=[$0], intKey=[$1])
> 00-07  Scan(groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/left.json, numFiles=1, columns=[`*`], 
> files=[maprfs:///tmp/left.json]]])
> {noformat}
> Data used in above queries
> {noformat}
> [root@centos-01 null_eq_joins]# cat left.json
> {
> "intKey" : 123,
> "bgintKey": 1234567,
> "strKey": "this is a test string",
> "boolKey": true,
> "fltKey": 123.786,
> "dblKey": 457.984,
> "timKey": "18:30:45",
> "dtKey": "1997-10-21",
> "tmstmpKey": "2007-04-30 13:10:02.047",
> "intrvldyKey": "P9DT38833S",
> "intrvlyrKey": "P255M"
> }
> [root@centos-01 null_equality_joins]#
> [root@centos-01 null_eq_joins]# cat right_all_nulls.json
> {
>  "intKey" : null,
>  "bgintKey": null,
>  "strKey": null,
>  "boolKey": null,
>  "fltKey": null,
>  "dblKey": null,
>  "timKey": null,
>  "dtKey": null,
>  "tmstmpKey": null,
>  "intrvldyKey": null,
>  "intrvlyrKey": null
> }
> [root@centos-01 null_eq_joins]#
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4842) SELECT * on JSON data results in NumberFormatException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651099#comment-15651099
 ] 

ASF GitHub Bot commented on DRILL-4842:
---

Github user Serhii-Harnyk commented on the issue:

https://github.com/apache/drill/pull/594
  
@chunhui-shi 
Results of performance test

| % of nullable records | with fix | master | %(t-t0)/t0 |
| 0.001 | 13986,8 | 13752,1 | 1,71% |
| 0.1 | 13873,6 | 13081,3 | 6,06% |
| 0.5 | 11345,6 | 7552,44 | 50,22% |
| 0.7 | 12699,1 | 5753,44 | 120,72% |
| 0.99 | 14544,2 | 494 | 2844,17% |

So increase of performance depends on % of nullable fields in dataset


> SELECT * on JSON data results in NumberFormatException
> --
>
> Key: DRILL-4842
> URL: https://issues.apache.org/jira/browse/DRILL-4842
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
> Attachments: tooManyNulls.json
>
>
> Note that doing SELECT c1 returns correct results, the failure is seen when 
> we do SELECT star. json.all_text_mode was set to true.
> JSON file tooManyNulls.json has one key c1 with 4096 nulls as its value and 
> the 4097th key c1 has the value "Hello World"
> git commit ID : aaf220ff
> MapR Drill 1.8.0 RPM
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.27 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT c1 FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> +--+
> |  c1  |
> +--+
> | Hello World  |
> +--+
> 1 row selected (0.243 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> Error: SYSTEM ERROR: NumberFormatException: Hello World
> Fragment 0:0
> [Error Id: 9cafb3f9-3d5c-478a-b55c-900602b8765e on centos-01.qa.lab:31010]
>  (java.lang.NumberFormatException) Hello World
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI():95
> 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt():120
> org.apache.drill.exec.test.generated.FiltererGen1169.doSetup():45
> org.apache.drill.exec.test.generated.FiltererGen1169.setup():54
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer():195
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema():107
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():78
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():257
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():251
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():251
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745