[jira] [Commented] (HIVE-24831) Support writing bloom filters in Parquet
[
https://issues.apache.org/jira/browse/HIVE-24831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291028#comment-17291028
]
Gabor Szadovszky commented on HIVE-24831:
-
See https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/README.md
for details of the required configuration. Search for {{bloom.filter}}.
> Support writing bloom filters in Parquet
>
>
> Key: HIVE-24831
> URL: https://issues.apache.org/jira/browse/HIVE-24831
> Project: Hive
> Issue Type: New Feature
> Components: Parquet
>Reporter: Csaba Ringhofer
>Priority: Major
>
> Parquet-mr 1.12.0 will add support for reading and writing Bloom filters.
> Reading doesn't need any action from Hive side, as it will be applied
> automatically if there is an Eq predicate on a column and the file contains a
> bloom filter.
> Writing needs some configuration, as Parquet-mr doesn't write bloom filters
> by default.
> Similar table properties could be used as in ORC, e.g.
> 'orc.bloom.filter.columns' to set the columns where Parquet-mr should write
> bloom filters. The same table property could be used by both Hive and Impala
> for the same purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-17843) UINT32 Parquet columns are handled as signed INT32-s, silently reading incorrect data
[
https://issues.apache.org/jira/browse/HIVE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364347#comment-16364347
]
Gabor Szadovszky commented on HIVE-17843:
-
I've created a parquet file but I am unable to upload/attach it.
> UINT32 Parquet columns are handled as signed INT32-s, silently reading
> incorrect data
> -
>
> Key: HIVE-17843
> URL: https://issues.apache.org/jira/browse/HIVE-17843
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Ivanfi
>Assignee: Janaki Lahorani
>Priority: Major
>
> An unsigned 32 bit Parquet column, such as
> {noformat}
> optional int32 uint_32_col (UINT_32)
> {noformat}
> is read by Hive as if it were signed, leading to incorrect results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15079) Hive cannot read Parquet string timetamps as TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171748#comment-16171748
]
Gabor Szadovszky commented on HIVE-15079:
-
Hi, [~kam338uk]. Unfortunately, I don't have time to work on Hive anymore. I
don't think it's fixed. Feel free to contribute ;)
> Hive cannot read Parquet string timetamps as TIMESTAMP data type
>
>
> Key: HIVE-15079
> URL: https://issues.apache.org/jira/browse/HIVE-15079
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sergio Peña
>
> The Hive Wiki for timestamps specifies that strings timestamps can be read by
> Hive.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-Timestamps
> {noformat}
> Supported conversions:
> Integer numeric types: Interpreted as UNIX timestamp in seconds
> Floating point numeric types: Interpreted as UNIX timestamp in seconds with
> decimal precision
> Strings: JDBC compliant java.sql.Timestamp format "-MM-DD
> HH:MM:SS.f" (9 decimal place precision)
> {noformat}
> This works fine with Text table formats, but when Parquet is used, then it
> throws the following exception:
> {noformat}
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> {noformat}
> How to reproduce
> {noformat}
> > create table t1 (id int, time string) stored as parquet;
> > insert into table t1 values (1,'2016-07-17 14:42:18');
> > alter table t1 replace columns (id int, time timestamp);
> > select * from t1
> {noformat}
> The above example will run fine if you use a TEXT format instead of PARQUET.
> This issue was raised on PARQUET-723
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-15079) Hive cannot read Parquet string timetamps as TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned HIVE-15079:
---
Assignee: (was: Gabor Szadovszky)
> Hive cannot read Parquet string timetamps as TIMESTAMP data type
>
>
> Key: HIVE-15079
> URL: https://issues.apache.org/jira/browse/HIVE-15079
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sergio Peña
>
> The Hive Wiki for timestamps specifies that strings timestamps can be read by
> Hive.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-Timestamps
> {noformat}
> Supported conversions:
> Integer numeric types: Interpreted as UNIX timestamp in seconds
> Floating point numeric types: Interpreted as UNIX timestamp in seconds with
> decimal precision
> Strings: JDBC compliant java.sql.Timestamp format "-MM-DD
> HH:MM:SS.f" (9 decimal place precision)
> {noformat}
> This works fine with Text table formats, but when Parquet is used, then it
> throws the following exception:
> {noformat}
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> {noformat}
> How to reproduce
> {noformat}
> > create table t1 (id int, time string) stored as parquet;
> > insert into table t1 values (1,'2016-07-17 14:42:18');
> > alter table t1 replace columns (id int, time timestamp);
> > select * from t1
> {noformat}
> The above example will run fine if you use a TEXT format instead of PARQUET.
> This issue was raised on PARQUET-723
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14714) Avoid misleading "java.io.IOException: Stream closed" when shutting down HoS
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509545#comment-15509545
]
Gabor Szadovszky commented on HIVE-14714:
-
Thanks a lot for reviewing/committing it. :)
> Avoid misleading "java.io.IOException: Stream closed" when shutting down HoS
>
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Fix For: 2.2.0
>
> Attachments: HIVE-14714.2.patch, HIVE-14714.3.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506460#comment-15506460
]
Gabor Szadovszky commented on HIVE-14714:
-
Review board link is removed as it is not relevant anymore. For this little
change, I would not create another one.
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.3.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Status: Open (was: Patch Available)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.3.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Status: Patch Available (was: Open)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.3.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Attachment: HIVE-14714.3.patch
Dropped the previous modifications and made only a little one based on
[~lirui]'s comments. Review it, please.
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.3.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503484#comment-15503484
]
Gabor Szadovszky commented on HIVE-14714:
-
Thanks a lot for the hint, [~lirui]. The fix of [HIVE-13895] should solve the
waiting problem.
However, in case of child.waitFor() is interrupted and the related process
still generates some output the IOException in the redirector threads would be
logged. (It might occur if the related spark configs are modified.) I think,
these exceptions might be misleading.
So, I would do a minimal modification to swallow these IOExceptions in case we
are about to stop the remote driver (isAlive is false). What do you think?
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502706#comment-15502706
]
Gabor Szadovszky commented on HIVE-14714:
-
Hi [~lirui],
# The root cause of the spark submit hang is that the refresh interval of the
checking of the process might set as large as it won't get the new state of the
remote driver in time. This value can be modified by the user therefore, I
would like to handle this situation.
# These threads are running in HS2 therefore, they won't be terminated in case
of beeline is closed. The only effect on the beeline is that it don't have to
wait for the timeout as the method stop() will return immediately. (In case of
HS2 is running in embedded mode, then these threads will be terminated but it
was the original behaviour which I haven't changed.)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495722#comment-15495722
]
Gabor Szadovszky commented on HIVE-14714:
-
Failing unit tests are not relevant.
Existing unit tests are covering the modified functionality or it is related to
logging which would quite hard to unit test. Manually tested (see review board
for details.)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493760#comment-15493760
]
Gabor Szadovszky commented on HIVE-14714:
-
The original problem was the listed exception and that beeline exited only
after 10s.
The root cause of the 10s delay was that in many cases the spark-submit process
does not end even in the case of the RemoteDriver has ended on the other side.
Therefore, the driverThread.join(1) really waits for 10s and then we are
interrupting it. Here comes the root cause of the logged exception. If we are
interrupting child.waitFor() the redirector threads gets IOExceptions in the
next readLine() as the related streams got closed.
I've redesigned the Redirector class therefore, it does not use any IO which
might hang the thread in case of interruption (e.g. BufferedReader.readLine()
cannot be interrupted, it waits for infinity if the related stream is open but
no input appears). After this redesign we are able to simply interrupt the
driver thread and let it keep working in the background until we have some
outputs to be gathered or the related timeout occurs. We do not have to hang
the client side to wait for all the threads to be finished.
Then came the unit test failure. The root cause was that protocol.endSession()
only sends a job via rpc asynchronously to close the session on the other side.
As there is no 10s delay anymore the unit tests executed each after another run
into the issue that the previous session is not closed properly. Therefore I've
implemented some trick make the end session synchronous.
Hope it describes my change properly and with my code comments makes it
understandable.
Any comments here or on the review board are more than welcome. :)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Status: Patch Available (was: In Progress)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Attachment: (was: HIVE-14714.2.patch)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Attachment: HIVE-14714.2.patch
Re-uploaded patch .2 with minor change (whitespace)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Status: In Progress (was: Patch Available)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Attachment: HIVE-14714.2.patch
Fix for the unit test failures.
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.2.patch, HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Status: Patch Available (was: In Progress)
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14714:
Attachment: HIVE-14714.patch
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14714.patch
>
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"
[
https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14714 started by Gabor Szadovszky.
---
> Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> ---
>
> Key: HIVE-14714
> URL: https://issues.apache.org/jira/browse/HIVE-14714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> After execute hive command with Spark, finishing the beeline session or
> even switch the engine causes IOException. The following executed Ctrl-D to
> finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
> the issue.
> From HS2 log:
> {code}
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote
> driver, interrupting...
> 2016-09-06 16:15:12,291 WARN org.apache.hive.spark.client.SparkClientImpl:
> [Driver]: Waiting thread interrupted, killing child process.
> 2016-09-06 16:15:12,296 WARN org.apache.hive.spark.client.SparkClientImpl:
> [stderr-redir-1]: Error in redirector thread.
> java.io.IOException: Stream closed
> at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14588) Add S3 credentials to the hidden configuration variable supported on HIVE-14207
[
https://issues.apache.org/jira/browse/HIVE-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14588:
Attachment: HIVE-14588.patch
Did not version the patch file as only minor changes made (removed trailing
whitespace).
> Add S3 credentials to the hidden configuration variable supported on
> HIVE-14207
> ---
>
> Key: HIVE-14588
> URL: https://issues.apache.org/jira/browse/HIVE-14588
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14588.patch
>
>
> Here's the list of S3 values we should hide:
> {code}
> fs.s3.awsAccessKeyId
> fs.s3.awsSecretAccessKey
> fs.s3n.awsAccessKeyId
> fs.s3n.awsSecretAccessKey
> fs.s3a.access.key
> fs.s3a.secret.key
> fs.s3a.proxy.password
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14588) Add S3 credentials to the hidden configuration variable supported on HIVE-14207
[
https://issues.apache.org/jira/browse/HIVE-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14588:
Attachment: (was: HIVE-14588.patch)
> Add S3 credentials to the hidden configuration variable supported on
> HIVE-14207
> ---
>
> Key: HIVE-14588
> URL: https://issues.apache.org/jira/browse/HIVE-14588
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Here's the list of S3 values we should hide:
> {code}
> fs.s3.awsAccessKeyId
> fs.s3.awsSecretAccessKey
> fs.s3n.awsAccessKeyId
> fs.s3n.awsSecretAccessKey
> fs.s3a.access.key
> fs.s3a.secret.key
> fs.s3a.proxy.password
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14588) Add S3 credentials to the hidden configuration variable supported on HIVE-14207
[
https://issues.apache.org/jira/browse/HIVE-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429260#comment-15429260
]
Gabor Szadovszky commented on HIVE-14588:
-
Failing tests are not related to this fix.
> Add S3 credentials to the hidden configuration variable supported on
> HIVE-14207
> ---
>
> Key: HIVE-14588
> URL: https://issues.apache.org/jira/browse/HIVE-14588
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14588.patch
>
>
> Here's the list of S3 values we should hide:
> {code}
> fs.s3.awsAccessKeyId
> fs.s3.awsSecretAccessKey
> fs.s3n.awsAccessKeyId
> fs.s3n.awsSecretAccessKey
> fs.s3a.access.key
> fs.s3a.secret.key
> fs.s3a.proxy.password
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14588) Add S3 credentials to the hidden configuration variable supported on HIVE-14207
[
https://issues.apache.org/jira/browse/HIVE-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14588:
Attachment: HIVE-14588.patch
Please, review my patch.
> Add S3 credentials to the hidden configuration variable supported on
> HIVE-14207
> ---
>
> Key: HIVE-14588
> URL: https://issues.apache.org/jira/browse/HIVE-14588
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14588.patch
>
>
> Here's the list of S3 values we should hide:
> {code}
> fs.s3.awsAccessKeyId
> fs.s3.awsSecretAccessKey
> fs.s3n.awsAccessKeyId
> fs.s3n.awsSecretAccessKey
> fs.s3a.access.key
> fs.s3a.secret.key
> fs.s3a.proxy.password
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14588) Add S3 credentials to the hidden configuration variable supported on HIVE-14207
[
https://issues.apache.org/jira/browse/HIVE-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14588:
Status: Patch Available (was: Open)
> Add S3 credentials to the hidden configuration variable supported on
> HIVE-14207
> ---
>
> Key: HIVE-14588
> URL: https://issues.apache.org/jira/browse/HIVE-14588
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14588.patch
>
>
> Here's the list of S3 values we should hide:
> {code}
> fs.s3.awsAccessKeyId
> fs.s3.awsSecretAccessKey
> fs.s3n.awsAccessKeyId
> fs.s3n.awsSecretAccessKey
> fs.s3a.access.key
> fs.s3a.secret.key
> fs.s3a.proxy.password
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-12884) NullPointerException in HiveParser.regularBody()
[
https://issues.apache.org/jira/browse/HIVE-12884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky resolved HIVE-12884.
-
Resolution: Cannot Reproduce
Closed as I was not able to reproduce. Feel free to reopen if reproduced or
additional information is available.
> NullPointerException in HiveParser.regularBody()
>
>
> Key: HIVE-12884
> URL: https://issues.apache.org/jira/browse/HIVE-12884
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.1.1
>Reporter: Bohumir Zamecnik
>Assignee: Gabor Szadovszky
> Attachments: HIVE-12884.q
>
>
> When I make a query like the following in Hive CLI I get a
> NullPointerException in HiveParser.regularBody().
> {code}
> create table some_table
> (
> day_timestamp bigint,
> guid_count bigint
> )
> row format delimited fields terminated by ',' stored as textfile;
> SET hive.merge.mapredfiles=true;
> SET mapreduce.input.fileinputformat.split.maxsize=5368709120;
> SET hivevar:tz_offset=8;
> SET hivevar:day_in_millis=8640;
> SET hivevar:year=2015;
> SET hivevar:month=02;
> SET hivevar:next_month=03;
> insert into table some_table
> select
> day_timestamp
> count(*) as guid_count
> from (
> select distinct
> guid,
> floor((`timestamp` / ${day_in_millis}) - ${tz_offset}) * ${day_in_millis}
> as day_timestamp,
> from source_table
> where year = ${year} and ((month = ${month}) or ((month = ${next_month})
> and (day = '01')))
> ) guids
> group by day_timestamp;
> {code}
> /tmp/username/hive.log:
> {code}
> 2016-01-18 10:05:40,505 ERROR [main]: ql.Driver
> (SessionState.java:printError(861)) - FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40975)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40183)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40059)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1519)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Hive 1.1.1 compiled from source with checksum
> c2d70ca009729fb13c073d599b4e5193.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395685#comment-15395685
]
Gabor Szadovszky commented on HIVE-14264:
-
Are you able to add some details about your environment? Unfortunately, I was
not able to figure anything out of the stacktrace.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14294) HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types
[
https://issues.apache.org/jira/browse/HIVE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14294:
Status: Open (was: Patch Available)
> HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into
> proper Parquet types
>
>
> Key: HIVE-14294
> URL: https://issues.apache.org/jira/browse/HIVE-14294
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 1.2.1
>Reporter: Cheng Lian
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14294.patch
>
>
> To reproduce this issue, run the following DDL:
> {code:sql}
> CREATE TABLE foo STORED AS PARQUET AS SELECT CAST(1 AS TINYINT);
> {code}
> And then check the schema of the written Parquet file:
> {noformat}
> $ parquet-schema $WAREHOUSE_PATH/foo/00_0
> message hive_schema {
> optional int32 _c0;
> }
> {noformat}
> When translating Hive types into Parquet types, {{TINYINT}} and {{SMALLINT}}
> should be translated into the {{int32 (INT_8)}} and {{int32 (INT_16)}}
> respectively. However, {{HiveSchemaConverter}} converts all of {{TINYINT}},
> {{SMALLINT}}, and {{INT}} into Parquet {{int32}}. This causes problem when
> accessing Parquet files generated by Hive in other systems since type
> information gets wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14294) HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types
[
https://issues.apache.org/jira/browse/HIVE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14294:
Status: Patch Available (was: Open)
> HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into
> proper Parquet types
>
>
> Key: HIVE-14294
> URL: https://issues.apache.org/jira/browse/HIVE-14294
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 1.2.1
>Reporter: Cheng Lian
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14294.patch
>
>
> To reproduce this issue, run the following DDL:
> {code:sql}
> CREATE TABLE foo STORED AS PARQUET AS SELECT CAST(1 AS TINYINT);
> {code}
> And then check the schema of the written Parquet file:
> {noformat}
> $ parquet-schema $WAREHOUSE_PATH/foo/00_0
> message hive_schema {
> optional int32 _c0;
> }
> {noformat}
> When translating Hive types into Parquet types, {{TINYINT}} and {{SMALLINT}}
> should be translated into the {{int32 (INT_8)}} and {{int32 (INT_16)}}
> respectively. However, {{HiveSchemaConverter}} converts all of {{TINYINT}},
> {{SMALLINT}}, and {{INT}} into Parquet {{int32}}. This causes problem when
> accessing Parquet files generated by Hive in other systems since type
> information gets wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14294) HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types
[
https://issues.apache.org/jira/browse/HIVE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14294:
Attachment: HIVE-14294.patch
> HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into
> proper Parquet types
>
>
> Key: HIVE-14294
> URL: https://issues.apache.org/jira/browse/HIVE-14294
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Cheng Lian
>Assignee: Gabor Szadovszky
> Attachments: HIVE-14294.patch
>
>
> To reproduce this issue, run the following DDL:
> {code:sql}
> CREATE TABLE foo STORED AS PARQUET AS SELECT CAST(1 AS TINYINT);
> {code}
> And then check the schema of the written Parquet file:
> {noformat}
> $ parquet-schema $WAREHOUSE_PATH/foo/00_0
> message hive_schema {
> optional int32 _c0;
> }
> {noformat}
> When translating Hive types into Parquet types, {{TINYINT}} and {{SMALLINT}}
> should be translated into the {{int32 (INT_8)}} and {{int32 (INT_16)}}
> respectively. However, {{HiveSchemaConverter}} converts all of {{TINYINT}},
> {{SMALLINT}}, and {{INT}} into Parquet {{int32}}. This causes problem when
> accessing Parquet files generated by Hive in other systems since type
> information gets wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14294) HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types
[
https://issues.apache.org/jira/browse/HIVE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-14294:
Target Version/s: 2.1.0
Status: Patch Available (was: Open)
Please, review the patch.
> HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into
> proper Parquet types
>
>
> Key: HIVE-14294
> URL: https://issues.apache.org/jira/browse/HIVE-14294
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 1.2.1
>Reporter: Cheng Lian
>Assignee: Gabor Szadovszky
>
> To reproduce this issue, run the following DDL:
> {code:sql}
> CREATE TABLE foo STORED AS PARQUET AS SELECT CAST(1 AS TINYINT);
> {code}
> And then check the schema of the written Parquet file:
> {noformat}
> $ parquet-schema $WAREHOUSE_PATH/foo/00_0
> message hive_schema {
> optional int32 _c0;
> }
> {noformat}
> When translating Hive types into Parquet types, {{TINYINT}} and {{SMALLINT}}
> should be translated into the {{int32 (INT_8)}} and {{int32 (INT_16)}}
> respectively. However, {{HiveSchemaConverter}} converts all of {{TINYINT}},
> {{SMALLINT}}, and {{INT}} into Parquet {{int32}}. This causes problem when
> accessing Parquet files generated by Hive in other systems since type
> information gets wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14294) HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types
[
https://issues.apache.org/jira/browse/HIVE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned HIVE-14294:
---
Assignee: Gabor Szadovszky
> HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into
> proper Parquet types
>
>
> Key: HIVE-14294
> URL: https://issues.apache.org/jira/browse/HIVE-14294
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Cheng Lian
>Assignee: Gabor Szadovszky
>
> To reproduce this issue, run the following DDL:
> {code:sql}
> CREATE TABLE foo STORED AS PARQUET AS SELECT CAST(1 AS TINYINT);
> {code}
> And then check the schema of the written Parquet file:
> {noformat}
> $ parquet-schema $WAREHOUSE_PATH/foo/00_0
> message hive_schema {
> optional int32 _c0;
> }
> {noformat}
> When translating Hive types into Parquet types, {{TINYINT}} and {{SMALLINT}}
> should be translated into the {{int32 (INT_8)}} and {{int32 (INT_16)}}
> respectively. However, {{HiveSchemaConverter}} converts all of {{TINYINT}},
> {{SMALLINT}}, and {{INT}} into Parquet {{int32}}. This causes problem when
> accessing Parquet files generated by Hive in other systems since type
> information gets wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12884) NullPointerException in HiveParser.regularBody()
[
https://issues.apache.org/jira/browse/HIVE-12884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated HIVE-12884:
Attachment: HIVE-12884.q
Actually, there are two mistakes in the query. One is mentioned above, the
other one is an extra ',' at the end of line "floor((`timestamp` ...".
Was not able to reproduce the NPE with any combinations of the mentioned
mistakes on the latest Hive 2.2.0. In any combinations ParseExceptions were
thrown.
See the attached query file created to reproduce the issue.
(After correcting the mentioned mistakes the query runs smoothly.)
> NullPointerException in HiveParser.regularBody()
>
>
> Key: HIVE-12884
> URL: https://issues.apache.org/jira/browse/HIVE-12884
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.1.1
>Reporter: Bohumir Zamecnik
>Assignee: Gabor Szadovszky
> Attachments: HIVE-12884.q
>
>
> When I make a query like the following in Hive CLI I get a
> NullPointerException in HiveParser.regularBody().
> {code}
> create table some_table
> (
> day_timestamp bigint,
> guid_count bigint
> )
> row format delimited fields terminated by ',' stored as textfile;
> SET hive.merge.mapredfiles=true;
> SET mapreduce.input.fileinputformat.split.maxsize=5368709120;
> SET hivevar:tz_offset=8;
> SET hivevar:day_in_millis=8640;
> SET hivevar:year=2015;
> SET hivevar:month=02;
> SET hivevar:next_month=03;
> insert into table some_table
> select
> day_timestamp
> count(*) as guid_count
> from (
> select distinct
> guid,
> floor((`timestamp` / ${day_in_millis}) - ${tz_offset}) * ${day_in_millis}
> as day_timestamp,
> from source_table
> where year = ${year} and ((month = ${month}) or ((month = ${next_month})
> and (day = '01')))
> ) guids
> group by day_timestamp;
> {code}
> /tmp/username/hive.log:
> {code}
> 2016-01-18 10:05:40,505 ERROR [main]: ql.Driver
> (SessionState.java:printError(861)) - FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40975)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40183)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40059)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1519)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Hive 1.1.1 compiled from source with checksum
> c2d70ca009729fb13c073d599b4e5193.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12884) NullPointerException in HiveParser.regularBody()
[
https://issues.apache.org/jira/browse/HIVE-12884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned HIVE-12884:
---
Assignee: Gabor Szadovszky
> NullPointerException in HiveParser.regularBody()
>
>
> Key: HIVE-12884
> URL: https://issues.apache.org/jira/browse/HIVE-12884
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.1.1
>Reporter: Bohumir Zamecnik
>Assignee: Gabor Szadovszky
>
> When I make a query like the following in Hive CLI I get a
> NullPointerException in HiveParser.regularBody().
> {code}
> create table some_table
> (
> day_timestamp bigint,
> guid_count bigint
> )
> row format delimited fields terminated by ',' stored as textfile;
> SET hive.merge.mapredfiles=true;
> SET mapreduce.input.fileinputformat.split.maxsize=5368709120;
> SET hivevar:tz_offset=8;
> SET hivevar:day_in_millis=8640;
> SET hivevar:year=2015;
> SET hivevar:month=02;
> SET hivevar:next_month=03;
> insert into table some_table
> select
> day_timestamp
> count(*) as guid_count
> from (
> select distinct
> guid,
> floor((`timestamp` / ${day_in_millis}) - ${tz_offset}) * ${day_in_millis}
> as day_timestamp,
> from source_table
> where year = ${year} and ((month = ${month}) or ((month = ${next_month})
> and (day = '01')))
> ) guids
> group by day_timestamp;
> {code}
> /tmp/username/hive.log:
> {code}
> 2016-01-18 10:05:40,505 ERROR [main]: ql.Driver
> (SessionState.java:printError(861)) - FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40975)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40183)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40059)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1519)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Hive 1.1.1 compiled from source with checksum
> c2d70ca009729fb13c073d599b4e5193.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384259#comment-15384259
]
Gabor Szadovszky commented on HIVE-14264:
-
Tried on 2.1.0-rc0 running on Derby db.
Beeline was started using the command: ./beeline --hiveconf
hive.cbo.enable=false -u jdbc:hive2://
Test data created using the following commands:
CREATE DATABASE hive_14264;
USE hive_14264;
CREATE TABLE table1 (key STRING, value STRING);
INSERT INTO TABLE table1
VALUES ('key1', 'value1'), (null, 'value2'), ('key3', null), (null,
null);
Tried to reproduce the issue by using the following queries:
0: jdbc:hive2://> SELECT * FROM table1 WHERE key IS NOT NULL;
OK
+-+---+--+
| table1.key | table1.value |
+-+---+--+
| key1| value1|
| key3| NULL |
+-+---+--+
2 rows selected (0.29 seconds)
0: jdbc:hive2://> SELECT * FROM table1 WHERE value IS NOT NULL;OK
+-+---+--+
| table1.key | table1.value |
+-+---+--+
| key1| value1|
| NULL| value2|
+-+---+--+
2 rows selected (0.087 seconds)
Queries executed as expected; issue was not reproducible.
Could you please provide more info to reproduce the issue?
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned HIVE-14264:
---
Assignee: Gabor Szadovszky
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
