date:20180602

[jira] [Commented] (DRILL-4364) Image Metadata Format Plugin

2018-06-02 Thread Akihiko Kusanagi (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499276#comment-16499276
 ] 

Akihiko Kusanagi commented on DRILL-4364:
-

I have updated the 
[document|https://gist.github.com/nagix/6cac019b7bec698a1b8a234a5268d09d] for 
this plugin.

> Image Metadata Format Plugin
> 
>
> Key: DRILL-4364
> URL: https://issues.apache.org/jira/browse/DRILL-4364
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Akihiko Kusanagi
>Assignee: Akihiko Kusanagi
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> Support querying of metadata in various image formats. This plugin leverages 
> [metadata-extractor|https://github.com/drewnoakes/metadata-extractor]. This 
> plugin is especially useful when querying on a large number of image files 
> stored in a distributed file system without building metadata repository in 
> advance.
> This plugin supports the following file formats.
>  * JPEG, TIFF, PSD, PNG, BMP, GIF, ICO, PCX, WAV, AVI, WebP, MOV, MP4, EPS
>  * Camera Raw: ARW (Sony), CRW/CR2 (Canon), NEF (Nikon), ORF (Olympus), RAF 
> (FujiFilm), RW2 (Panasonic), RWL (Leica), SRW (Samsung), X3F (Foveon)
> This plugin enables to read the following metadata.
>  * Exif, IPTC, XMP, JFIF / JFXX, ICC Profiles, Photoshop fields, PNG 
> properties, BMP properties, GIF properties, ICO properties, PCX properties, 
> WAV properties, AVI properties, WebP properties, QuickTime properties, MP4 
> properties, EPS properties
> Since each type of metadata has a different set of fields, the plugin returns 
> a set of commonly-used fields such as the image width, height and bits per 
> pixels for ease of use.
> *Examples:*
> Querying on a JPEG file with the property descriptive: true
> {noformat}
> 0: jdbc:drill:zk=local> select FileName, * from 
> dfs.`4349313028_f69ffa0257_o.jpg`;
> +--+--+--+++-+--+--+---++---+--+--++---++-+-+--+--+--++--+-+---+---+--+-+--+
> | FileName | FileSize | FileDateTime | Format | PixelWidth | PixelHeight | 
> BitsPerPixel | DPIWidth | DPIHeight | Orientaion | ColorMode | HasAlpha | 
> Duration | VideoCodec | FrameRate | AudioCodec | AudioSampleSize | 
> AudioSampleRate | JPEG | JFIF | ExifIFD0 | ExifSubIFD | Interoperability | 
> GPS | ExifThumbnail | Photoshop | IPTC | Huffman | FileType |
> +--+--+--+++-+--+--+---++---+--+--++---++-+-+--+--+--++--+-+---+---+--+-+--+
> | 4349313028_f69ffa0257_o.jpg | 257213 bytes | Fri Mar 09 12:09:34 +08:00 
> 2018 | JPEG | 1199 | 800 | 24 | 96 | 96 | Unknown (0) | RGB | false | 
> 00:00:00 | Unknown | 0 | Unknown | 0 | 0 | 
> {"CompressionType":"Baseline","DataPrecision":"8 bits","ImageHeight":"800 
> pixels","ImageWidth":"1199 pixels","NumberOfComponents":"3","Component1":"Y 
> component: Quantization table 0, Sampling factors 2 horiz/2 
> vert","Component2":"Cb component: Quantization table 1, Sampling factors 1 
> horiz/1 vert","Component3":"Cr component: Quantization table 1, Sampling 
> factors 1 horiz/1 vert"} | 
> {"Version":"1.1","ResolutionUnits":"inch","XResolution":"96 
> dots","YResolution":"96 
> dots","ThumbnailWidthPixels":"0","ThumbnailHeightPixels":"0"} | 
> {"Software":"Picasa 3.0"} | 
> {"ExifVersion":"2.10","UniqueImageID":"d65e93b836d15a0c5e041e6b7258c76e"} | 
> {"InteroperabilityIndex":"Unknown ()","InteroperabilityVersion":"1.00"} | 
> {"GPSVersionID":".022","GPSLatitudeRef":"N","GPSLatitude":"47° 32' 
> 15.98\"","GPSLongitudeRef":"W","GPSLongitude":"-122° 2' 
> 6.37\"","GPSAltitudeRef":"Sea level","GPSAltitude":"0 metres"} | 
> {"Compression":"JPEG (old-style)","XResolution":"72 dots per 
> inch","YResolution":"72 dots per 
> inch","ResolutionUnit":"Inch","ThumbnailOffset":"414 
> bytes","ThumbnailLength":"7213 bytes"} | {} | 
> {"Keywords":"135;2002;issaquah;police car;wa;washington"} | 
> {"NumberOfTables":"4 Huffman tables"} | 
> {"DetectedFileTypeName":"JPEG","DetectedFileTypeLongName":"Joint Photographic 
> Experts 
> Group","DetectedMIMEType":"image/jpeg","ExpectedFileNameExtension":"jpg"} |
>

[jira] [Commented] (DRILL-4364) Image Metadata Format Plugin

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499274#comment-16499274
 ] 

ASF GitHub Bot commented on DRILL-4364:
---

nagix commented on issue #367: DRILL-4364: Image Metadata Format Plugin
URL: https://github.com/apache/drill/pull/367#issuecomment-374920364
 
 
   I have updated the dependent library and rebased the code. Now, Image 
Metadata Format Plugin supports:
   - Querying metadata in JPEG, TIFF, PSD, PNG, BMP, GIF, ICO, PCX, WAV, AVI, 
WebP, MOV, MP4, EPS, ARW (Sony), CRW/CR2 (Canon), NEF (Nikon), ORF (Olympus), 
RAF (FujiFilm), RW2 (Panasonic), RWL (Leica), SRW (Samsung) and X3F (Foveon) 
files
   - Retrieval of both a human-readable descriptive string and a 
machine-readable typed value from each tag
   - A set of commonly-used fields such as the image width, height and bits per 
pixels for ease of use
   - Querying directories of image files to create your own metadata DB
   
   See also the original JIRA 
([DRILL-4364](https://issues.apache.org/jira/browse/DRILL-4364)) and the 
[document](https://gist.github.com/nagix/6cac019b7bec698a1b8a234a5268d09d).
   
   @cgivre, @kkhatua, please can you review this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Image Metadata Format Plugin
> 
>
> Key: DRILL-4364
> URL: https://issues.apache.org/jira/browse/DRILL-4364
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Akihiko Kusanagi
>Assignee: Akihiko Kusanagi
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> Support querying of metadata in various image formats. This plugin leverages 
> [metadata-extractor|https://github.com/drewnoakes/metadata-extractor]. This 
> plugin is especially useful when querying on a large number of image files 
> stored in a distributed file system without building metadata repository in 
> advance.
> This plugin supports the following file formats.
>  * JPEG, TIFF, PSD, PNG, BMP, GIF, ICO, PCX, WAV, AVI, WebP, MOV, MP4, EPS
>  * Camera Raw: ARW (Sony), CRW/CR2 (Canon), NEF (Nikon), ORF (Olympus), RAF 
> (FujiFilm), RW2 (Panasonic), RWL (Leica), SRW (Samsung), X3F (Foveon)
> This plugin enables to read the following metadata.
>  * Exif, IPTC, XMP, JFIF / JFXX, ICC Profiles, Photoshop fields, PNG 
> properties, BMP properties, GIF properties, ICO properties, PCX properties, 
> WAV properties, AVI properties, WebP properties, QuickTime properties, MP4 
> properties, EPS properties
> Since each type of metadata has a different set of fields, the plugin returns 
> a set of commonly-used fields such as the image width, height and bits per 
> pixels for ease of use.
> *Examples:*
> Querying on a JPEG file with the property descriptive: true
> {noformat}
> 0: jdbc:drill:zk=local> select FileName, * from 
> dfs.`4349313028_f69ffa0257_o.jpg`;
> +--+--+--+++-+--+--+---++---+--+--++---++-+-+--+--+--++--+-+---+---+--+-+--+
> | FileName | FileSize | FileDateTime | Format | PixelWidth | PixelHeight | 
> BitsPerPixel | DPIWidth | DPIHeight | Orientaion | ColorMode | HasAlpha | 
> Duration | VideoCodec | FrameRate | AudioCodec | AudioSampleSize | 
> AudioSampleRate | JPEG | JFIF | ExifIFD0 | ExifSubIFD | Interoperability | 
> GPS | ExifThumbnail | Photoshop | IPTC | Huffman | FileType |
> +--+--+--+++-+--+--+---++---+--+--++---++-+-+--+--+--++--+-+---+---+--+-+--+
> | 4349313028_f69ffa0257_o.jpg | 257213 bytes | Fri Mar 09 12:09:34 +08:00 
> 2018 | JPEG | 1199 | 800 | 24 | 96 | 96 | Unknown (0) | RGB | false | 
> 00:00:00 | Unknown | 0 | Unknown | 0 | 0 | 
> {"CompressionType":"Baseline","DataPrecision":"8 bits","ImageHeight":"800 
> pixels","ImageWidth":"1199 pixels","NumberOfComponents":"3","Component1":"Y 
> component: Quantization table 0, Sampling factors 2 horiz/2 
> vert","Component2":"Cb component: Quantization table 1, Sampling factors 1 
> horiz/1 vert","Component3":"Cr component: Quantization table 1, Sampling 
> factors 1 horiz/1

[jira] [Commented] (DRILL-4020) The not-equal operator returns incorrect results when used on the HBase row key

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499240#comment-16499240
 ] 

ASF GitHub Bot commented on DRILL-4020:
---

parthchandra commented on issue #309: DRILL-4020: The not-equal operator 
returns incorrect results when used on the HBase row key
URL: https://github.com/apache/drill/pull/309#issuecomment-394126791
 
 
   Reopened. Unit test is failing and so are some functional tests 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> The not-equal operator returns incorrect results when used on the HBase row 
> key
> ---
>
> Key: DRILL-4020
> URL: https://issues.apache.org/jira/browse/DRILL-4020
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0
> Environment: Drill Sandbox
>Reporter: Akihiko Kusanagi
>Priority: Critical
>
> Create a test HBase table:
> {noformat}
> hbase> create 'table', 'f'
> hbase> put 'table', 'row1', 'f:c', 'value1'
> hbase> put 'table', 'row2', 'f:c', 'value2'
> hbase> put 'table', 'row3', 'f:c', 'value3'
> {noformat}
> The table looks like this:
> {noformat}
> 0: jdbc:drill:zk=maprdemo:5181> SELECT CONVERT_FROM(row_key, 'UTF8') FROM 
> hbase.`table`;
> +-+
> | EXPR$0  |
> +-+
> | row1|
> | row2|
> | row3|
> +-+
> 1 row selected (4.596 seconds)
> {noformat}
> However, this query returns incorrect results when a not-equal operator is 
> used on the row key:
> {noformat}
> 0: jdbc:drill:zk=maprdemo:5181> SELECT CONVERT_FROM(row_key, 'UTF8') FROM 
> hbase.`table` WHERE row_key <> 'row1';
> +-+
> | EXPR$0  |
> +-+
> | row1|
> | row2|
> | row3|
> +-+
> 1 row selected (0.573 seconds)
> {noformat}
> In the query plan, there is no RowFilter:
> {noformat}
> 00-00Screen
> 00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
> [tableName=table, startRow=, stopRow=, filter=null], columns=[`row_key`]]])
> {noformat}
> When the query has multiple not-equal operators, it works fine:
> {noformat}
> 0: jdbc:drill:zk=maprdemo:5181> SELECT CONVERT_FROM(row_key, 'UTF8') FROM 
> hbase.`table` WHERE row_key <> 'row1' AND row_key <> 'row2';
> +-+
> | EXPR$0  |
> +-+
> | row3|
> +-+
> 1 row selected (0.255 seconds)
> {noformat}
> In the query plan, a FilterList has two RowFilters with NOT_EQUAL operators:
> {noformat}
> 00-00Screen
> 00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
> [tableName=table, startRow=, stopRow=, filter=FilterList AND (2/2): 
> [RowFilter (NOT_EQUAL, row1), RowFilter (NOT_EQUAL, row2)]], 
> columns=[`row_key`]]])
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4020) The not-equal operator returns incorrect results when used on the HBase row key

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499239#comment-16499239
 ] 

ASF GitHub Bot commented on DRILL-4020:
---

nagix opened a new pull request #309: DRILL-4020: The not-equal operator 
returns incorrect results when used on the HBase row key
URL: https://github.com/apache/drill/pull/309
 
 
   - Added a condition that checks if the filter to the scan specification 
doesn't have NOT_EQUAL operator
   - Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> The not-equal operator returns incorrect results when used on the HBase row 
> key
> ---
>
> Key: DRILL-4020
> URL: https://issues.apache.org/jira/browse/DRILL-4020
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0
> Environment: Drill Sandbox
>Reporter: Akihiko Kusanagi
>Priority: Critical
>
> Create a test HBase table:
> {noformat}
> hbase> create 'table', 'f'
> hbase> put 'table', 'row1', 'f:c', 'value1'
> hbase> put 'table', 'row2', 'f:c', 'value2'
> hbase> put 'table', 'row3', 'f:c', 'value3'
> {noformat}
> The table looks like this:
> {noformat}
> 0: jdbc:drill:zk=maprdemo:5181> SELECT CONVERT_FROM(row_key, 'UTF8') FROM 
> hbase.`table`;
> +-+
> | EXPR$0  |
> +-+
> | row1|
> | row2|
> | row3|
> +-+
> 1 row selected (4.596 seconds)
> {noformat}
> However, this query returns incorrect results when a not-equal operator is 
> used on the row key:
> {noformat}
> 0: jdbc:drill:zk=maprdemo:5181> SELECT CONVERT_FROM(row_key, 'UTF8') FROM 
> hbase.`table` WHERE row_key <> 'row1';
> +-+
> | EXPR$0  |
> +-+
> | row1|
> | row2|
> | row3|
> +-+
> 1 row selected (0.573 seconds)
> {noformat}
> In the query plan, there is no RowFilter:
> {noformat}
> 00-00Screen
> 00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
> [tableName=table, startRow=, stopRow=, filter=null], columns=[`row_key`]]])
> {noformat}
> When the query has multiple not-equal operators, it works fine:
> {noformat}
> 0: jdbc:drill:zk=maprdemo:5181> SELECT CONVERT_FROM(row_key, 'UTF8') FROM 
> hbase.`table` WHERE row_key <> 'row1' AND row_key <> 'row2';
> +-+
> | EXPR$0  |
> +-+
> | row3|
> +-+
> 1 row selected (0.255 seconds)
> {noformat}
> In the query plan, a FilterList has two RowFilters with NOT_EQUAL operators:
> {noformat}
> 00-00Screen
> 00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
> [tableName=table, startRow=, stopRow=, filter=FilterList AND (2/2): 
> [RowFilter (NOT_EQUAL, row1), RowFilter (NOT_EQUAL, row2)]], 
> columns=[`row_key`]]])
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499238#comment-16499238
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

parthchandra commented on issue #1201: DRILL-4091: Support for additional gis 
operations in gis contrib module
URL: https://github.com/apache/drill/pull/1201#issuecomment-394126688
 
 
   It's committed into master.
   
   On Sat, Jun 2, 2018 at 4:46 AM, Chris Sandison 
   wrote:
   
   > @parthchandra  @cgivre
   >  why has this been closed?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>Assignee: Karol Potocki
>Priority: Major
>  Labels: ready-to-commit
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6353) Upgrade Parquet MR dependencies

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499236#comment-16499236
 ] 

ASF GitHub Bot commented on DRILL-6353:
---

vrozov commented on issue #1259: DRILL-6353: Upgrade Parquet MR dependencies
URL: https://github.com/apache/drill/pull/1259#issuecomment-394126286
 
 
   @parthchandra @arina-ielchiieva The PR is ready for the final review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6463) ProfileParser cannot parse costs when using MockScanBatch

2018-06-02 Thread Gautam Kumar Parai (JIRA)

Gautam Kumar Parai created DRILL-6463:
-

 Summary: ProfileParser cannot parse costs when using MockScanBatch
 Key: DRILL-6463
 URL: https://issues.apache.org/jira/browse/DRILL-6463
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Gautam Kumar Parai
Assignee: Gautam Kumar Parai
 Fix For: 1.14.0


One of the unit testHashAggrSecondaryTertiarySpill() runs into this issue 
although the issue is generic. It happens due to cost being stored in an int 
which overflows with big enough rows/data size and becomes negative. This 
causes the Profile parser to error out on seeing negative costs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499020#comment-16499020
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

ChrisSandison commented on issue #1201: DRILL-4091: Support for additional gis 
operations in gis contrib module
URL: https://github.com/apache/drill/pull/1201#issuecomment-394081247
 
 
   @parthchandra @cgivre why has this been closed?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>Assignee: Karol Potocki
>Priority: Major
>  Labels: ready-to-commit
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

2018-06-02 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499018#comment-16499018
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

ChrisSandison commented on issue #1201: DRILL-4091: Support for additional gis 
operations in gis contrib module
URL: https://github.com/apache/drill/pull/1201#issuecomment-394081247
 
 
   @parthchandra @cgivre why has this been closed?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>Assignee: Karol Potocki
>Priority: Major
>  Labels: ready-to-commit
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6293) Unable to read hive(2.1.1) tables using Drill 1.13.0

2018-06-02 Thread Vitalii Diravka (JIRA)



 [ 
https://issues.apache.org/jira/browse/DRILL-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka reassigned DRILL-6293:
--

Assignee: (was: Vitalii Diravka)

> Unable to read hive(2.1.1) tables using Drill 1.13.0 
> -
>
> Key: DRILL-6293
> URL: https://issues.apache.org/jira/browse/DRILL-6293
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anup Tiwari
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: hive-metastore-2.1.1.jar
>
>
> Hi,
> {color:#22}I am not able to read my hive tables in drill 1.13.0 and with 
> same plugin conf it was working in Drill 1.12.0 and 1.10.0.{color}
>   
>  *Hive Plugin :-*
> {code:java}
>  {
>    "type": "hive",
>    "enabled": true,
>    "configProps":
>{
>  "hive.metastore.uris": "thrift://prod-hadoop-1xx.com:9083", 
>   "hive.metastore.sasl.enabled": "false",   
>   "fs.default.name": "hdfs://prod-hadoop-1xx.com:9000"   
>}
> } 
>   {code}
> *Query :-* 
>  select id from hive.cad where log_date = '2018-03-18' limit 3;
>   
>  *Error :-* 
> {code}
> 2018-03-20 14:25:27,351 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] INFO 
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 254f337f-9ac3-b66f-ed17-1de459da3283: select id from hive.cad where log_date 
> = '2018-03-18' limit 3
>  2018-03-20 14:25:27,354 [254f337f-9ac3-b66f-ed17-1de459da3283:foreman] WARN 
> o.a.d.e.s.h.DrillHiveMetaStoreClient - Failure while attempting to get hive 
> table. Retries once.
>  org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
>   at 
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111) 
> ~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
>   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) 
> ~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
>  ~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
>  ~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1344)
>  ~[drill-hive-exec-shaded-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.getHiveReadEntryHelper(DrillHiveMetaStoreClient.java:285)
>  ~[drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:535)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:531)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  [guava-18.0.jar:na]
>   at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> [guava-18.0.jar:na]
>   at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  [guava-18.0.jar:na]
>   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> [guava-18.0.jar:na]
>   at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
> [guava-18.0.jar:na]
>   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
> [guava-18.0.jar:na]
>   at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
>  [guava-18.0.jar:na]
>   at 
> org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getHiveReadEntry(DrillHiveMetaStoreClient.java:495)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSelectionBaseOnName(HiveSchemaFactory.java:233)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getDrillTable(HiveSchemaFactory.java:213)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.schema.HiveDatabaseSchema.getTable(HiveDatabaseSchema.java:62)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getTable(HiveSchemaFactory.java:201)
>  [drill-storage-hive-core-1.13.0.jar:1.13.0]
>   at 
> org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:82)
>  [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
>   at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:257) 
> [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
>   at 
>

[jira] [Assigned] (DRILL-6460) Different hive-metastore jar files in accordance with the version of Hive

2018-06-02 Thread Vitalii Diravka (JIRA)



 [ 
https://issues.apache.org/jira/browse/DRILL-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka reassigned DRILL-6460:
--

Assignee: (was: Vitalii Diravka)

> Different hive-metastore jar files in accordance with the version of Hive
> -
>
> Key: DRILL-6460
> URL: https://issues.apache.org/jira/browse/DRILL-6460
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Hive
>Affects Versions: 1.13.0
>Reporter: Vitalii Diravka
>Priority: Major
>
> Hive metastore client can't guarantee the proper work with other versions of 
> Hive. So it should be exactly the same version of Hive metastore client and 
> server.
> In Spark the user can specify the path, where hive-metastore jar is placed. 
> For example, if spark works with Hive2.1.1 version, it is necessary to put 
> the path to hive-metastore-2.1.1.jar for the property 
> _*spark.sql.hive.metastore.jars*_ 
>  
> [http://spark.apache.org/docs/latest/sql-programming-guide.html#interacting-with-different-versions-of-hive-metastore]
> The similar way should be implemented in Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4364) Image Metadata Format Plugin

[jira] [Commented] (DRILL-4364) Image Metadata Format Plugin

[jira] [Commented] (DRILL-4020) The not-equal operator returns incorrect results when used on the HBase row key

[jira] [Commented] (DRILL-4020) The not-equal operator returns incorrect results when used on the HBase row key

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[jira] [Commented] (DRILL-6353) Upgrade Parquet MR dependencies

[jira] [Created] (DRILL-6463) ProfileParser cannot parse costs when using MockScanBatch

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[jira] [Assigned] (DRILL-6293) Unable to read hive(2.1.1) tables using Drill 1.13.0

[jira] [Assigned] (DRILL-6460) Different hive-metastore jar files in accordance with the version of Hive

11 matches

Site Navigation

Mail list logo

Footer information