[jira] [Commented] (HIVE-16767) Update people website with recent changes
[ https://issues.apache.org/jira/browse/HIVE-16767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026321#comment-16026321 ] Jimmy Xiang commented on HIVE-16767: [~lirui], my info is right. Thanks > Update people website with recent changes > - > > Key: HIVE-16767 > URL: https://issues.apache.org/jira/browse/HIVE-16767 > Project: Hive > Issue Type: Task > Components: Documentation >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-16767.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16429: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks Zhihai for the patch. Pushed to the master branch. > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16429.000.patch, HIVE-16429.001.patch > > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16433) Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-16433:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks Zhihai for the patch. Pushed to the master branch.
> Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
> ---
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
> currently {{rj}} is set to null in ExecDriver.shutdown which is called from
> other thread for query cancellation. It can happen at any time. There is a
> potential race condition, the {{rj}} is still accessed after shutdown is
> called. For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying {{rj}} is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag {{jobKilled}} to make sure {{rj.killJob()}} will be only
> called once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16433) Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968060#comment-15968060
]
Jimmy Xiang commented on HIVE-16433:
Ok, thanks for the explanation. +1
> Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
> ---
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
> currently {{rj}} is set to null in ExecDriver.shutdown which is called from
> other thread for query cancellation. It can happen at any time. There is a
> potential race condition, the {{rj}} is still accessed after shutdown is
> called. For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying {{rj}} is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag {{jobKilled}} to make sure {{rj.killJob()}} will be only
> called once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968052#comment-15968052 ] Jimmy Xiang commented on HIVE-16429: Thanks for making the change. +1 > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16429.000.patch, HIVE-16429.001.patch > > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16433) Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967948#comment-15967948
]
Jimmy Xiang commented on HIVE-16433:
By the way, is the new method isTaskShutdown used any place?
> Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
> ---
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
> currently {{rj}} is set to null in ExecDriver.shutdown which is called from
> other thread for query cancellation. It can happen at any time. There is a
> potential race condition, the {{rj}} is still accessed after shutdown is
> called. For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying {{rj}} is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag {{jobKilled}} to make sure {{rj.killJob()}} will be only
> called once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16433) Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967940#comment-15967940
]
Jimmy Xiang commented on HIVE-16433:
Can you attach a stacktrace to show the NPE problem?
> Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
> ---
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
> currently {{rj}} is set to null in ExecDriver.shutdown which is called from
> other thread for query cancellation. It can happen at any time. There is a
> potential race condition, the {{rj}} is still accessed after shutdown is
> called. For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying {{rj}} is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag {{jobKilled}} to make sure {{rj.killJob()}} will be only
> called once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967932#comment-15967932 ] Jimmy Xiang commented on HIVE-16429: Please add a new function which takes two more parameters so that the change is less. > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16429.000.patch > > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15630) add operation handle before operation.run instead of after operation.run
[
https://issues.apache.org/jira/browse/HIVE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-15630:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks Zhihai for the patch. Pushed to the master branch.
> add operation handle before operation.run instead of after operation.run
>
>
> Key: HIVE-15630
> URL: https://issues.apache.org/jira/browse/HIVE-15630
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15630.000.patch, HIVE-15630.001.patch
>
>
> Add operation handle before operation.run instead of after operation.run. So
> when session is closed, all the running operations from {{operation.run}} can
> also be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15630) add operation handle before operation.run instead of after operation.run
[
https://issues.apache.org/jira/browse/HIVE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959457#comment-15959457
]
Jimmy Xiang commented on HIVE-15630:
Cool. +1
> add operation handle before operation.run instead of after operation.run
>
>
> Key: HIVE-15630
> URL: https://issues.apache.org/jira/browse/HIVE-15630
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15630.000.patch, HIVE-15630.001.patch
>
>
> Add operation handle before operation.run instead of after operation.run. So
> when session is closed, all the running operations from {{operation.run}} can
> also be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16348) HoS query is canceled but error message shows RPC is closed
[ https://issues.apache.org/jira/browse/HIVE-16348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16348: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) The test failures are not related. Thanks Xuefu for the review. Pushed to the master branch. > HoS query is canceled but error message shows RPC is closed > --- > > Key: HIVE-16348 > URL: https://issues.apache.org/jira/browse/HIVE-16348 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-16348.1.patch > > > When a HoS query is interrupted in getting app id, it keeps trying to get > status till timedout, and return some RPC is closed error message which is > misleading. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16348) HoS query is canceled but error message shows RPC is closed
[ https://issues.apache.org/jira/browse/HIVE-16348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16348: --- Status: Patch Available (was: Open) > HoS query is canceled but error message shows RPC is closed > --- > > Key: HIVE-16348 > URL: https://issues.apache.org/jira/browse/HIVE-16348 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-16348.1.patch > > > When a HoS query is interrupted in getting app id, it keeps trying to get > status till timedout, and return some RPC is closed error message which is > misleading. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16348) HoS query is canceled but error message shows RPC is closed
[ https://issues.apache.org/jira/browse/HIVE-16348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16348: --- Attachment: HIVE-16348.1.patch > HoS query is canceled but error message shows RPC is closed > --- > > Key: HIVE-16348 > URL: https://issues.apache.org/jira/browse/HIVE-16348 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-16348.1.patch > > > When a HoS query is interrupted in getting app id, it keeps trying to get > status till timedout, and return some RPC is closed error message which is > misleading. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16348) HoS query is canceled but error message shows RPC is closed
[ https://issues.apache.org/jira/browse/HIVE-16348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HIVE-16348: -- > HoS query is canceled but error message shows RPC is closed > --- > > Key: HIVE-16348 > URL: https://issues.apache.org/jira/browse/HIVE-16348 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > > When a HoS query is interrupted in getting app id, it keeps trying to get > status till timedout, and return some RPC is closed error message which is > misleading. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16292) SparkUtilities upload to HDFS doesn't work with viewfs
[
https://issues.apache.org/jira/browse/HIVE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-16292:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks Rui for the review. Pushed to the master branch.
> SparkUtilities upload to HDFS doesn't work with viewfs
> --
>
> Key: HIVE-16292
> URL: https://issues.apache.org/jira/browse/HIVE-16292
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-16292.1.patch, HIVE-16292.2.patch
>
>
> If scratchdir is set to viewfs, HoS fails with exception like
> {noformt}
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar,
> expected: hdfs://nameservice1
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16292) SparkUtilities upload to HDFS doesn't work with viewfs
[
https://issues.apache.org/jira/browse/HIVE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-16292:
---
Attachment: HIVE-16292.2.patch
Patch 2, fixed one more issue.
> SparkUtilities upload to HDFS doesn't work with viewfs
> --
>
> Key: HIVE-16292
> URL: https://issues.apache.org/jira/browse/HIVE-16292
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-16292.1.patch, HIVE-16292.2.patch
>
>
> If scratchdir is set to viewfs, HoS fails with exception like
> {noformt}
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar,
> expected: hdfs://nameservice1
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16292) SparkUtilities upload to HDFS doesn't work with viewfs
[
https://issues.apache.org/jira/browse/HIVE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-16292:
---
Attachment: HIVE-16292.1.patch
> SparkUtilities upload to HDFS doesn't work with viewfs
> --
>
> Key: HIVE-16292
> URL: https://issues.apache.org/jira/browse/HIVE-16292
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-16292.1.patch
>
>
> If scratchdir is set to viewfs, HoS fails with exception like
> {noformt}
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar,
> expected: hdfs://nameservice1
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16292) SparkUtilities upload to HDFS doesn't work with viewfs
[
https://issues.apache.org/jira/browse/HIVE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-16292:
---
Status: Patch Available (was: Open)
> SparkUtilities upload to HDFS doesn't work with viewfs
> --
>
> Key: HIVE-16292
> URL: https://issues.apache.org/jira/browse/HIVE-16292
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-16292.1.patch
>
>
> If scratchdir is set to viewfs, HoS fails with exception like
> {noformt}
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar,
> expected: hdfs://nameservice1
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16292) SparkUtilities upload to HDFS doesn't work with viewfs
[
https://issues.apache.org/jira/browse/HIVE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang reassigned HIVE-16292:
--
> SparkUtilities upload to HDFS doesn't work with viewfs
> --
>
> Key: HIVE-16292
> URL: https://issues.apache.org/jira/browse/HIVE-16292
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
>
> If scratchdir is set to viewfs, HoS fails with exception like
> {noformt}
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar,
> expected: hdfs://nameservice1
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16286: --- Attachment: HIVE-16286.1.patch > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: HIVE-16286.1.patch > > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HIVE-16286: -- > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16286: --- Status: Patch Available (was: Open) > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: HIVE-16286.1.patch > > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15915) Emit progress percentage in getting operation status
[ https://issues.apache.org/jira/browse/HIVE-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15915: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Those test failures are not related. Thanks Xuefu for the review. Integrated into the master branch. > Emit progress percentage in getting operation status > > > Key: HIVE-15915 > URL: https://issues.apache.org/jira/browse/HIVE-15915 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15915.1.patch > > > When running a query asynchronously, client may want to check the progress > periodically. HIVE-15473 is to support progressing bar on beeline for Tez. > For this issue, we just want the progress percentage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15915) Emit progress percentage in getting operation status
[ https://issues.apache.org/jira/browse/HIVE-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15915: --- Status: Patch Available (was: Open) Attached a patch that emits progress percentage for Hive on both MR and Spark. > Emit progress percentage in getting operation status > > > Key: HIVE-15915 > URL: https://issues.apache.org/jira/browse/HIVE-15915 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-15915.1.patch > > > When running a query asynchronously, client may want to check the progress > periodically. HIVE-15473 is to support progressing bar on beeline for Tez. > For this issue, we just want the progress percentage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15915) Emit progress percentage in getting operation status
[ https://issues.apache.org/jira/browse/HIVE-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15915: --- Attachment: HIVE-15915.1.patch > Emit progress percentage in getting operation status > > > Key: HIVE-15915 > URL: https://issues.apache.org/jira/browse/HIVE-15915 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-15915.1.patch > > > When running a query asynchronously, client may want to check the progress > periodically. HIVE-15473 is to support progressing bar on beeline for Tez. > For this issue, we just want the progress percentage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15915) Emit progress percentage in getting operation status
[ https://issues.apache.org/jira/browse/HIVE-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HIVE-15915: -- > Emit progress percentage in getting operation status > > > Key: HIVE-15915 > URL: https://issues.apache.org/jira/browse/HIVE-15915 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > > When running a query asynchronously, client may want to check the progress > periodically. HIVE-15473 is to support progressing bar on beeline for Tez. > For this issue, we just want the progress percentage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout
[
https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864186#comment-15864186
]
Jimmy Xiang commented on HIVE-15671:
+1
> RPCServer.registerClient() erroneously uses server/client handshake timeout
> for connection timeout
> --
>
> Key: HIVE-15671
> URL: https://issues.apache.org/jira/browse/HIVE-15671
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15671.1.patch, HIVE-15671.patch
>
>
> {code}
> /**
>* Tells the RPC server to expect a connection from a new client.
>* ...
>*/
> public Future registerClient(final String clientId, String secret,
> RpcDispatcher serverDispatcher) {
> return registerClient(clientId, secret, serverDispatcher,
> config.getServerConnectTimeoutMs());
> }
> {code}
> {{config.getServerConnectTimeoutMs()}} returns value for
> *hive.spark.client.server.connect.timeout*, which is meant for timeout for
> handshake between Hive client and remote Spark driver. Instead, the timeout
> should be *hive.spark.client.connect.timeout*, which is for timeout for
> remote Spark driver in connecting back to Hive client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15630) add operation handle before operation.run instead of after operation.run
[
https://issues.apache.org/jira/browse/HIVE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859954#comment-15859954
]
Jimmy Xiang commented on HIVE-15630:
For async operation, this change should be fine. However, for sync operation,
with this change, if a query takes some time, could it be killed due to
time-out?
> add operation handle before operation.run instead of after operation.run
>
>
> Key: HIVE-15630
> URL: https://issues.apache.org/jira/browse/HIVE-15630
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15630.000.patch
>
>
> Add operation handle before operation.run instead of after operation.run. So
> when session is closed, all the running operations from {{operation.run}} can
> also be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15485) Investigate the DoAs failure in HoS
[
https://issues.apache.org/jira/browse/HIVE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843082#comment-15843082
]
Jimmy Xiang commented on HIVE-15485:
Thanks for making the change. Looks good to me. +1
> Investigate the DoAs failure in HoS
> ---
>
> Key: HIVE-15485
> URL: https://issues.apache.org/jira/browse/HIVE-15485
> Project: Hive
> Issue Type: Bug
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-15485.1.patch, HIVE-15485.patch
>
>
> With DoAs enabled, HoS failed with following errors:
> {code}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> systest tries to renew a token with renewer hive
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:484)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:7543)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:555)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewDelegationToken(AuthorizationProviderProxyClientProtocol.java:674)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:999)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2141)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1783)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)
> {code}
> It is related to the change from HIVE-14383. It looks like that SparkSubmit
> logs in Kerberos with passed in hive principal/keytab and then tries to
> create a hdfs delegation token for user systest with renewer hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15485) Investigate the DoAs failure in HoS
[
https://issues.apache.org/jira/browse/HIVE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840465#comment-15840465
]
Jimmy Xiang commented on HIVE-15485:
For doAs, add kinit etc to the beginning of the list; for the other add
principal etc at the end. If you are concerned with performance, will
LinkedList be better than ArrayList here?
By the way, should keyTabFile + ";" be two argvs?
> Investigate the DoAs failure in HoS
> ---
>
> Key: HIVE-15485
> URL: https://issues.apache.org/jira/browse/HIVE-15485
> Project: Hive
> Issue Type: Bug
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-15485.patch
>
>
> With DoAs enabled, HoS failed with following errors:
> {code}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> systest tries to renew a token with renewer hive
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:484)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:7543)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:555)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewDelegationToken(AuthorizationProviderProxyClientProtocol.java:674)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:999)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2141)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1783)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)
> {code}
> It is related to the change from HIVE-14383. It looks like that SparkSubmit
> logs in Kerberos with passed in hive principal/keytab and then tries to
> create a hdfs delegation token for user systest with renewer hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15485) Investigate the DoAs failure in HoS
[
https://issues.apache.org/jira/browse/HIVE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840049#comment-15840049
]
Jimmy Xiang commented on HIVE-15485:
Could you put the two changes in your patch in the same place to make it a
little easier to understand?
> Investigate the DoAs failure in HoS
> ---
>
> Key: HIVE-15485
> URL: https://issues.apache.org/jira/browse/HIVE-15485
> Project: Hive
> Issue Type: Bug
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-15485.patch
>
>
> With DoAs enabled, HoS failed with following errors:
> {code}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> systest tries to renew a token with renewer hive
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:484)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:7543)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:555)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewDelegationToken(AuthorizationProviderProxyClientProtocol.java:674)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:999)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2141)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1783)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)
> {code}
> It is related to the change from HIVE-14383. It looks like that SparkSubmit
> logs in Kerberos with passed in hive principal/keytab and then tries to
> create a hdfs delegation token for user systest with renewer hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15629) Set DDLTask’s exception with its subtask’s exception
[ https://issues.apache.org/jira/browse/HIVE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15629: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Zhihai for the patch. > Set DDLTask’s exception with its subtask’s exception > > > Key: HIVE-15629 > URL: https://issues.apache.org/jira/browse/HIVE-15629 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15629.000.patch > > > Set DDLTask’s exception with its subtask’s exception, So the exception from > subtask in DDLTask can be propagated to TaskRunner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15629) Set DDLTask’s exception with its subtask’s exception
[ https://issues.apache.org/jira/browse/HIVE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826604#comment-15826604 ] Jimmy Xiang commented on HIVE-15629: +1 > Set DDLTask’s exception with its subtask’s exception > > > Key: HIVE-15629 > URL: https://issues.apache.org/jira/browse/HIVE-15629 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-15629.000.patch > > > Set DDLTask’s exception with its subtask’s exception, So the exception from > subtask in DDLTask can be propagated to TaskRunner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15564) set task's jobID with hadoop map reduce job ID for PartialScanTask, MergeFileTask and ColumnTruncateTask.
[
https://issues.apache.org/jira/browse/HIVE-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-15564:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to the master branch. The test failures are not related. Thanks Zhihai
for the patch.
> set task's jobID with hadoop map reduce job ID for PartialScanTask,
> MergeFileTask and ColumnTruncateTask.
> -
>
> Key: HIVE-15564
> URL: https://issues.apache.org/jira/browse/HIVE-15564
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15564.000.patch, HIVE-15564.001.patch
>
>
> set task's jobID with hadoop map reduce job ID for PartialScanTask,
> MergeFileTask and ColumnTruncateTask. Since ExecDriver already set task's
> jobID with hadoop map reduce job ID, this patch will make it consistent for
> all tasks who submit map reduce job.
> {code}
> // Finally SUBMIT the JOB!
> rj = jc.submitJob(job);
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15563) Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.
[
https://issues.apache.org/jira/browse/HIVE-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-15563:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to the master branch. The test failures are not related. Thanks Zhihai
for the patch.
> Ignore Illegal Operation state transition exception in SQLOperation.runQuery
> to expose real exception.
> --
>
> Key: HIVE-15563
> URL: https://issues.apache.org/jira/browse/HIVE-15563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15563.000.patch, HIVE-15563.001.patch
>
>
> Ignore Illegal Operation state transition exception in SQLOperation.runQuery
> to expose real exception.
> setState may create Illegal Operation state transition exception which may
> hide the real exception. we see the following exception happened from
> {{setState(OperationState.ERROR);}} in SQLOperation.runQuery
> {code}
> org.apache.hive.service.cli.operation.Operation: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Illegal Operation state
> transition from CLOSED to ERROR
> at
> org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:91)
> at
> org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:97)
> at
> org.apache.hive.service.cli.operation.Operation.setState(Operation.java:154)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:241)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:82)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:288)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:301)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15563) Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.
[
https://issues.apache.org/jira/browse/HIVE-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816399#comment-15816399
]
Jimmy Xiang commented on HIVE-15563:
Thanks for making the change. Looks good to me. +1
> Ignore Illegal Operation state transition exception in SQLOperation.runQuery
> to expose real exception.
> --
>
> Key: HIVE-15563
> URL: https://issues.apache.org/jira/browse/HIVE-15563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15563.000.patch, HIVE-15563.001.patch
>
>
> Ignore Illegal Operation state transition exception in SQLOperation.runQuery
> to expose real exception.
> setState may create Illegal Operation state transition exception which may
> hide the real exception. we see the following exception happened from
> {{setState(OperationState.ERROR);}} in SQLOperation.runQuery
> {code}
> org.apache.hive.service.cli.operation.Operation: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Illegal Operation state
> transition from CLOSED to ERROR
> at
> org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:91)
> at
> org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:97)
> at
> org.apache.hive.service.cli.operation.Operation.setState(Operation.java:154)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:241)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:82)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:288)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:301)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15564) set task's jobID with hadoop map reduce job ID for PartialScanTask, MergeFileTask and ColumnTruncateTask.
[
https://issues.apache.org/jira/browse/HIVE-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816219#comment-15816219
]
Jimmy Xiang commented on HIVE-15564:
+1. Thanks for the explanation.
> set task's jobID with hadoop map reduce job ID for PartialScanTask,
> MergeFileTask and ColumnTruncateTask.
> -
>
> Key: HIVE-15564
> URL: https://issues.apache.org/jira/browse/HIVE-15564
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15564.000.patch, HIVE-15564.001.patch
>
>
> set task's jobID with hadoop map reduce job ID for PartialScanTask,
> MergeFileTask and ColumnTruncateTask. Since ExecDriver already set task's
> jobID with hadoop map reduce job ID, this patch will make it consistent for
> all tasks who submit map reduce job.
> {code}
> // Finally SUBMIT the JOB!
> rj = jc.submitJob(job);
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15563) Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.
[
https://issues.apache.org/jira/browse/HIVE-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815852#comment-15815852
]
Jimmy Xiang commented on HIVE-15563:
Instead of catching the state transition exception, should we check the state
before set the new state the same way as HiveSQLException is handled, and log
the exception:
{noformat}
if ((getStatus().getState() == OperationState.CANCELED)
|| (getStatus().getState() == OperationState.TIMEDOUT)
|| (getStatus().getState() == OperationState.CLOSED)) {
return;
}
{noformat}
> Ignore Illegal Operation state transition exception in SQLOperation.runQuery
> to expose real exception.
> --
>
> Key: HIVE-15563
> URL: https://issues.apache.org/jira/browse/HIVE-15563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15563.000.patch
>
>
> Ignore Illegal Operation state transition exception in SQLOperation.runQuery
> to expose real exception.
> setState may create Illegal Operation state transition exception which may
> hide the real exception. we see the following exception happened from
> {{setState(OperationState.ERROR);}} in SQLOperation.runQuery
> {code}
> org.apache.hive.service.cli.operation.Operation: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Illegal Operation state
> transition from CLOSED to ERROR
> at
> org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:91)
> at
> org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:97)
> at
> org.apache.hive.service.cli.operation.Operation.setState(Operation.java:154)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:241)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:82)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:288)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:301)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15564) set task's jobID with hadoop map reduce job ID for PartialScanTask, MergeFileTask and ColumnTruncateTask.
[
https://issues.apache.org/jira/browse/HIVE-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815696#comment-15815696
]
Jimmy Xiang commented on HIVE-15564:
For these tasks, the job id is set in the final block. It is not set the same
way as other tasks, is this because these tasks don't call
updateStatusInQueryDisplay()?
> set task's jobID with hadoop map reduce job ID for PartialScanTask,
> MergeFileTask and ColumnTruncateTask.
> -
>
> Key: HIVE-15564
> URL: https://issues.apache.org/jira/browse/HIVE-15564
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-15564.000.patch
>
>
> set task's jobID with hadoop map reduce job ID for PartialScanTask,
> MergeFileTask and ColumnTruncateTask. Since ExecDriver already set task's
> jobID with hadoop map reduce job ID, this patch will make it consistent for
> all tasks who submit map reduce job.
> {code}
> // Finally SUBMIT the JOB!
> rj = jc.submitJob(job);
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15470) Catch Throwable instead of Exception in driver.execute.
[ https://issues.apache.org/jira/browse/HIVE-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15470: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Zhihai for the patch. > Catch Throwable instead of Exception in driver.execute. > --- > > Key: HIVE-15470 > URL: https://issues.apache.org/jira/browse/HIVE-15470 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15470.000.patch > > > Catch Throwable instead of Exception in driver.execute. So the failed query > with Throwable not Exception will also be logged and reported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15470) Catch Throwable instead of Exception in driver.execute.
[ https://issues.apache.org/jira/browse/HIVE-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764570#comment-15764570 ] Jimmy Xiang commented on HIVE-15470: +1 > Catch Throwable instead of Exception in driver.execute. > --- > > Key: HIVE-15470 > URL: https://issues.apache.org/jira/browse/HIVE-15470 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-15470.000.patch > > > Catch Throwable instead of Exception in driver.execute. So the failed query > with Throwable not Exception will also be logged and reported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15252) hive.security.command.whitelist doesn't work for 'reload function'
[
https://issues.apache.org/jira/browse/HIVE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685097#comment-15685097
]
Jimmy Xiang commented on HIVE-15252:
+1
> hive.security.command.whitelist doesn't work for 'reload function'
> --
>
> Key: HIVE-15252
> URL: https://issues.apache.org/jira/browse/HIVE-15252
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HIVE-15252.1.patch
>
>
> Right now the 'reload function' is handled specially in {{HiveCommand#find}}:
> {code}
> return null;
> } else if(command.length > 1 && "reload".equalsIgnoreCase(command[0])
> && "function".equalsIgnoreCase(command[1])) {
> //special handling for SQL "reload function"
> return null;
> }
> {code}
> However, this means the {{hive.security.command.whitelist}} doesn't work for
> the command. It's better to first do the privilege check and then the rest.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15208) Query string should be HTML encoded for Web UI
[ https://issues.apache.org/jira/browse/HIVE-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15208: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to the master branch. Thanks Xuefu for the review. > Query string should be HTML encoded for Web UI > -- > > Key: HIVE-15208 > URL: https://issues.apache.org/jira/browse/HIVE-15208 > Project: Hive > Issue Type: Bug > Components: Web UI >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15208.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15208) Query string should be HTML encoded for Web UI
[ https://issues.apache.org/jira/browse/HIVE-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15208: --- Status: Patch Available (was: Open) Submitted a patch that HTML-escapes the query string. > Query string should be HTML encoded for Web UI > -- > > Key: HIVE-15208 > URL: https://issues.apache.org/jira/browse/HIVE-15208 > Project: Hive > Issue Type: Bug > Components: Web UI >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-15208.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15208) Query string should be HTML encoded for Web UI
[ https://issues.apache.org/jira/browse/HIVE-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-15208: --- Attachment: HIVE-15208.1.patch > Query string should be HTML encoded for Web UI > -- > > Key: HIVE-15208 > URL: https://issues.apache.org/jira/browse/HIVE-15208 > Project: Hive > Issue Type: Bug > Components: Web UI >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-15208.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14984) Hive-WebUI access results in Request is a replay (34) attack
[ https://issues.apache.org/jira/browse/HIVE-14984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592987#comment-15592987 ] Jimmy Xiang commented on HIVE-14984: Good. Thanks. > Hive-WebUI access results in Request is a replay (34) attack > > > Key: HIVE-14984 > URL: https://issues.apache.org/jira/browse/HIVE-14984 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Venkat Sambath >Assignee: Barna Zsombor Klara > Attachments: HIVE-14984.patch > > > When trying to access kerberized webui of HS2, The following error is received > GSSException: Failure unspecified at GSS-API level (Mechanism level: Request > is a replay (34)) > While this is not happening for RM webui (checked if kerberos webui is > enabled) > To reproduce the issue > Try running > curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt > http://:10002/ > from any cluster nodes > or > Try accessing the URL from a VM with windows machine and firefox browser to > replicate the issue > The following workaround helped, but need a permanent solution for the bug > Workaround: > = > First access the index.html directly and then actual URL of webui > curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt > http://:10002/index.html > curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt > http://:10002 > In browser: > First access > http://:10002/index.html > then > http://:10002 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14928) Analyze table no scan mess up schema
[ https://issues.apache.org/jira/browse/HIVE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14928: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Integrated to master branch. Thanks Prasanth for the review. > Analyze table no scan mess up schema > > > Key: HIVE-14928 > URL: https://issues.apache.org/jira/browse/HIVE-14928 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 2.2.0 > > Attachments: HIVE-14928.1.patch, HIVE-14928.2.patch > > > StatsNoJobTask uses static variables partUpdates and table to track stats > changes. If multiple analyze no scan tasks run at the same time, then > table/partition schema could mess up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14928) Analyze table no scan mess up schema
[ https://issues.apache.org/jira/browse/HIVE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14928: --- Status: Patch Available (was: Open) > Analyze table no scan mess up schema > > > Key: HIVE-14928 > URL: https://issues.apache.org/jira/browse/HIVE-14928 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-14928.1.patch, HIVE-14928.2.patch > > > StatsNoJobTask uses static variables partUpdates and table to track stats > changes. If multiple analyze no scan tasks run at the same time, then > table/partition schema could mess up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14928) Analyze table no scan mess up schema
[ https://issues.apache.org/jira/browse/HIVE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14928: --- Attachment: HIVE-14928.2.patch Rebased the patch to master latest. > Analyze table no scan mess up schema > > > Key: HIVE-14928 > URL: https://issues.apache.org/jira/browse/HIVE-14928 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-14928.1.patch, HIVE-14928.2.patch > > > StatsNoJobTask uses static variables partUpdates and table to track stats > changes. If multiple analyze no scan tasks run at the same time, then > table/partition schema could mess up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14928) Analyze table no scan mess up schema
[ https://issues.apache.org/jira/browse/HIVE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14928: --- Status: Open (was: Patch Available) > Analyze table no scan mess up schema > > > Key: HIVE-14928 > URL: https://issues.apache.org/jira/browse/HIVE-14928 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-14928.1.patch, HIVE-14928.2.patch > > > StatsNoJobTask uses static variables partUpdates and table to track stats > changes. If multiple analyze no scan tasks run at the same time, then > table/partition schema could mess up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14928) Analyze table no scan mess up schema
[ https://issues.apache.org/jira/browse/HIVE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14928: --- Status: Patch Available (was: Open) > Analyze table no scan mess up schema > > > Key: HIVE-14928 > URL: https://issues.apache.org/jira/browse/HIVE-14928 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang > Attachments: HIVE-14928.1.patch > > > StatsNoJobTask uses static variables partUpdates and table to track stats > changes. If multiple analyze no scan tasks run at the same time, then > table/partition schema could mess up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14928) Analyze table no scan mess up schema
[ https://issues.apache.org/jira/browse/HIVE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14928: --- Attachment: HIVE-14928.1.patch > Analyze table no scan mess up schema > > > Key: HIVE-14928 > URL: https://issues.apache.org/jira/browse/HIVE-14928 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang > Attachments: HIVE-14928.1.patch > > > StatsNoJobTask uses static variables partUpdates and table to track stats > changes. If multiple analyze no scan tasks run at the same time, then > table/partition schema could mess up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511503#comment-15511503 ] Jimmy Xiang commented on HIVE-14774: +1 > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14697) Can not access kerberized HS2 Web UI
[
https://issues.apache.org/jira/browse/HIVE-14697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460225#comment-15460225
]
Jimmy Xiang commented on HIVE-14697:
+1
> Can not access kerberized HS2 Web UI
>
>
> Key: HIVE-14697
> URL: https://issues.apache.org/jira/browse/HIVE-14697
> Project: Hive
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.1.0
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-14697.patch
>
>
> Failed to access kerberized HS2 WebUI with following error msg:
> {code}
> curl -v -u : --negotiate http://util185.phx2.cbsig.net:10002/
> > GET / HTTP/1.1
> > Host: util185.phx2.cbsig.net:10002
> > Authorization: Negotiate YIIU7...[redacted]...
> > User-Agent: curl/7.42.1
> > Accept: */*
> >
> < HTTP/1.1 413 FULL head
> < Content-Length: 0
> < Connection: close
> < Server: Jetty(7.6.0.v20120127)
> {code}
> It is because the Jetty default request header (4K) is too small in some
> kerberos case.
> So this patch is to increase the request header to 64K.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14368: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Zhihai for the patch. > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405254#comment-15405254 ] Jimmy Xiang commented on HIVE-14368: +1 > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399979#comment-15399979 ] Jimmy Xiang commented on HIVE-14340: Multiple configurations/interfaces should be fine. > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399753#comment-15399753 ] Jimmy Xiang commented on HIVE-14340: Query hook is a little ambiguous. Since post execution hooks are already supported, does this mean we just need to add pre compilation hook? Isn't pre semantic analysis good for this purpose? > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14331) Task should set exception for failed map reduce job.
[ https://issues.apache.org/jira/browse/HIVE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14331: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Zhihai for the patch. > Task should set exception for failed map reduce job. > > > Key: HIVE-14331 > URL: https://issues.apache.org/jira/browse/HIVE-14331 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.1.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14331.000.patch > > > Task should set exception for failed map reduce job. So the exception can be > seen in HookContext. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14331) Task should set exception for failed map reduce job.
[ https://issues.apache.org/jira/browse/HIVE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392946#comment-15392946 ] Jimmy Xiang commented on HIVE-14331: +1 > Task should set exception for failed map reduce job. > > > Key: HIVE-14331 > URL: https://issues.apache.org/jira/browse/HIVE-14331 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.1.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-14331.000.patch > > > Task should set exception for failed map reduce job. So the exception can be > seen in HookContext. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14258) Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too long to finish without reporting progress
[
https://issues.apache.org/jira/browse/HIVE-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381071#comment-15381071
]
Jimmy Xiang commented on HIVE-14258:
+1
> Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too
> long to finish without reporting progress
> ---
>
> Key: HIVE-14258
> URL: https://issues.apache.org/jira/browse/HIVE-14258
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-14258.patch
>
>
> Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too
> long to finish without reporting progress.
> This timeout happened when reducer.close() is called in ReduceTask.java.
> CommonJoinOperator.genUniqueJoinObject() called by reducer.close() will loop
> over every row in the AbstractRowContainer. This can take a long time if
> there are a large number or rows, and during this time, it does not report
> progress. If this runs for long enough more than "mapreduce.task.timeout",
> ApplicationMaster will kill the task for failing to report progress.
> we configured "mapreduce.task.timeout" as 10 minutes. I captured the stack
> trace in the 10 minutes before AM killed the reduce task at 2016-07-15
> 07:19:11.
> The following three stack traces can prove it:
> at 2016-07-15 07:09:42:
> {code}
> "main" prio=10 tid=0x7f90ec017000 nid=0xd193 runnable [0x7f90f62e5000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:154)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x0007deecefb0> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:252)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:214)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:232)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007deecb978> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
> at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2359)
> - locked <0x0007deec8f70> (a
> org.apache.hadoop.io.SequenceFile$Reader)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2491)
> - locked <0x0007deec8f70> (a
> org.apache.hadoop.io.SequenceFile$Reader)
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)
> - locked <0x0007deec82f0> (a
> org.apache.hadoop.mapred.SequenceFileRecordReader)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.next(RowContainer.java:267)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.next(RowContainer.java:74)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:750)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
[jira] [Commented] (HIVE-14207) Strip HiveConf hidden params in webui conf
[ https://issues.apache.org/jira/browse/HIVE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371149#comment-15371149 ] Jimmy Xiang commented on HIVE-14207: Other than the test speed, it is also to avoid port conflicts in case multiple tests run in parallel. > Strip HiveConf hidden params in webui conf > -- > > Key: HIVE-14207 > URL: https://issues.apache.org/jira/browse/HIVE-14207 > Project: Hive > Issue Type: Bug > Components: Web UI >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14207.patch > > > HIVE-12338 introduced a new web ui, which has a page that displays the > current HiveConf being used by HS2. However, before it displays that config, > it does not strip entries from it which are considered "hidden" conf > parameters, thus exposing those values from a web-ui for HS2. We need to add > stripping to this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13960) Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for back-to-back synchronous operations.
[
https://issues.apache.org/jira/browse/HIVE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-13960:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Integrated into trunk. Thanks Zhihai for the patch.
> Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for
> back-to-back synchronous operations.
>
>
> Key: HIVE-13960
> URL: https://issues.apache.org/jira/browse/HIVE-13960
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.2.0
>
> Attachments: HIVE-13960.000.patch
>
>
> Session timeout may happen before
> HIVE_SERVER2_IDLE_SESSION_TIMEOUT(hive.server2.idle.session.timeout) for
> back-to-back synchronous operations.
> This issue can happen with the following two operations op1 and op2: op2 is a
> synchronous long running operation, op2 is running right after op1 is closed.
>
> 1. closeOperation(op1) is called:
> this will set {{lastIdleTime}} with value System.currentTimeMillis() because
> {{opHandleSet}} becomes empty after {{closeOperation}} remove op1 from
> {{opHandleSet}}.
> 2. op2 is running for long time by calling {{executeStatement}} right after
> closeOperation(op1) is called.
> If op2 is running for more than HIVE_SERVER2_IDLE_SESSION_TIMEOUT, then the
> session will timeout even when op2 is still running.
> We hit this issue when we use PyHive to execute non-async operation
> The following is the exception we see:
> {code}
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 126, in
> close
> _check_status(response)
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 362, in
> _check_status
> raise OperationalError(response)
> OperationalError: TCloseSessionResp(status=TStatus(errorCode=0,
> errorMessage='Session does not exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not
> exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13900) HiveStatement.executeAsync() may not work properly when hive.server2.async.exec.async.compile is turned on
[ https://issues.apache.org/jira/browse/HIVE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325930#comment-15325930 ] Jimmy Xiang commented on HIVE-13900: +1 > HiveStatement.executeAsync() may not work properly when > hive.server2.async.exec.async.compile is turned on > -- > > Key: HIVE-13900 > URL: https://issues.apache.org/jira/browse/HIVE-13900 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13900.1.patch > > > HIVE-13882 handles HiveStatement.executeQuery() when > hive.server2.async.exec.async.compile is turned on. Notice we may also have > similar issue when executeAsync() is called. Investigate what would be the > good approach for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13960) Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for back-to-back synchronous operations.
[
https://issues.apache.org/jira/browse/HIVE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319671#comment-15319671
]
Jimmy Xiang commented on HIVE-13960:
Cool. +1.
> Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for
> back-to-back synchronous operations.
>
>
> Key: HIVE-13960
> URL: https://issues.apache.org/jira/browse/HIVE-13960
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-13960.000.patch
>
>
> Session timeout may happen before
> HIVE_SERVER2_IDLE_SESSION_TIMEOUT(hive.server2.idle.session.timeout) for
> back-to-back synchronous operations.
> This issue can happen with the following two operations op1 and op2: op2 is a
> synchronous long running operation, op2 is running right after op1 is closed.
>
> 1. closeOperation(op1) is called:
> this will set {{lastIdleTime}} with value System.currentTimeMillis() because
> {{opHandleSet}} becomes empty after {{closeOperation}} remove op1 from
> {{opHandleSet}}.
> 2. op2 is running for long time by calling {{executeStatement}} right after
> closeOperation(op1) is called.
> If op2 is running for more than HIVE_SERVER2_IDLE_SESSION_TIMEOUT, then the
> session will timeout even when op2 is still running.
> We hit this issue when we use PyHive to execute non-async operation
> The following is the exception we see:
> {code}
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 126, in
> close
> _check_status(response)
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 362, in
> _check_status
> raise OperationalError(response)
> OperationalError: TCloseSessionResp(status=TStatus(errorCode=0,
> errorMessage='Session does not exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not
> exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13960) Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for back-to-back synchronous operations.
[
https://issues.apache.org/jira/browse/HIVE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318851#comment-15318851
]
Jimmy Xiang commented on HIVE-13960:
Good to set lastIdleTime to 0 at acquire(). Why do we need to check both
opHandleSet.isEmpty() and pendingCount = 0? When will these two don't match?
> Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for
> back-to-back synchronous operations.
>
>
> Key: HIVE-13960
> URL: https://issues.apache.org/jira/browse/HIVE-13960
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-13960.000.patch
>
>
> Session timeout may happen before
> HIVE_SERVER2_IDLE_SESSION_TIMEOUT(hive.server2.idle.session.timeout) for
> back-to-back synchronous operations.
> This issue can happen with the following two operations op1 and op2: op2 is a
> synchronous long running operation, op2 is running right after op1 is closed.
>
> 1. closeOperation(op1) is called:
> this will set {{lastIdleTime}} with value System.currentTimeMillis() because
> {{opHandleSet}} becomes empty after {{closeOperation}} remove op1 from
> {{opHandleSet}}.
> 2. op2 is running for long time by calling {{executeStatement}} right after
> closeOperation(op1) is called.
> If op2 is running for more than HIVE_SERVER2_IDLE_SESSION_TIMEOUT, then the
> session will timeout even when op2 is still running.
> We hit this issue when we use PyHive to execute non-async operation
> The following is the exception we see:
> {code}
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 126, in
> close
> _check_status(response)
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 362, in
> _check_status
> raise OperationalError(response)
> OperationalError: TCloseSessionResp(status=TStatus(errorCode=0,
> errorMessage='Session does not exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not
> exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"
[
https://issues.apache.org/jira/browse/HIVE-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305978#comment-15305978
]
Jimmy Xiang commented on HIVE-13882:
Thanks. +1
> When hive.server2.async.exec.async.compile is turned on, from JDBC we will
> get "The query did not generate a result set"
> -
>
> Key: HIVE-13882
> URL: https://issues.apache.org/jira/browse/HIVE-13882
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13882.1.patch, HIVE-13882.2.patch
>
>
> The following would fail with "The query did not generate a result set"
> stmt.execute("SET hive.driver.parallel.compilation=true");
> stmt.execute("SET hive.server2.async.exec.async.compile=true");
> ResultSet res = stmt.executeQuery("SELECT * FROM " + tableName);
> res.next();
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"
[
https://issues.apache.org/jira/browse/HIVE-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304758#comment-15304758
]
Jimmy Xiang commented on HIVE-13882:
[~aihuaxu], yeah, I was thinking to file a followup issue. Thanks a lot for the
patch. Can we also add a test to make sure jdbc works when
hive.driver.parallel.compilation=false and
hive.server2.async.exec.async.compile=true?
> When hive.server2.async.exec.async.compile is turned on, from JDBC we will
> get "The query did not generate a result set"
> -
>
> Key: HIVE-13882
> URL: https://issues.apache.org/jira/browse/HIVE-13882
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13882.1.patch
>
>
> The following would fail with "The query did not generate a result set"
> stmt.execute("SET hive.driver.parallel.compilation=true");
> stmt.execute("SET hive.server2.async.exec.async.compile=true");
> ResultSet res = stmt.executeQuery("SELECT * FROM " + tableName);
> res.next();
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300677#comment-15300677
]
Jimmy Xiang commented on HIVE-13149:
Thanks for the explanation. +1
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch,
> HIVE-13149.3.patch, HIVE-13149.4.patch, HIVE-13149.5.patch,
> HIVE-13149.6.patch, HIVE-13149.7.patch, HIVE-13149.8.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300328#comment-15300328
]
Jimmy Xiang commented on HIVE-13149:
In your fix to TestJdbcWithMiniHS2, you removed the setup method, instead, you
create the connection in each test. Any special reason?
Another thing, hs2conn is not set to null after the close. So the teardown
closes the connection for the second time, right?
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch,
> HIVE-13149.3.patch, HIVE-13149.4.patch, HIVE-13149.5.patch,
> HIVE-13149.6.patch, HIVE-13149.7.patch, HIVE-13149.8.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.1 Status: Resolved (was: Patch Available) Integrated into trunk. Thans Xuefu and Lefty for the review. > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.0.1 > > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294581#comment-15294581 ] Jimmy Xiang commented on HIVE-13782: [~leftylev], any comments on the new parameter description above? > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294579#comment-15294579
]
Jimmy Xiang commented on HIVE-13813:
Looks good to me too. +1
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293449#comment-15293449 ] Jimmy Xiang commented on HIVE-13782: By result information, I meant if the query will return some resultset or not. Some query doesn't return anything, for example, insert some value to a table. How about we change it to something like below? "Whether to enable compiling async query asynchronously. If enabled, it is unknown if the query will have any resultset before compilation completed." > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Status: Patch Available (was: Open) > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Attachment: HIVE-13782.1.patch > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Priority: Minor (was: Major) > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13679) Pass diagnostic message to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13679: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Integrated to trunk. Thanks Aihua for the review. > Pass diagnostic message to failure hooks > > > Key: HIVE-13679 > URL: https://issues.apache.org/jira/browse/HIVE-13679 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13679.1.patch > > > Pass diagnostic message to failure hooks. This is useful for debugging remote > job failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13679) Pass diagnostic message to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13679: --- Status: Patch Available (was: Open) > Pass diagnostic message to failure hooks > > > Key: HIVE-13679 > URL: https://issues.apache.org/jira/browse/HIVE-13679 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13679.1.patch > > > Pass diagnostic message to failure hooks. This is useful for debugging remote > job failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13679) Pass diagnostic message to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13679: --- Attachment: HIVE-13679.1.patch > Pass diagnostic message to failure hooks > > > Key: HIVE-13679 > URL: https://issues.apache.org/jira/browse/HIVE-13679 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13679.1.patch > > > Pass diagnostic message to failure hooks. This is useful for debugging remote > job failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13559) Pass exception to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13559: --- Status: Patch Available (was: Open) > Pass exception to failure hooks > --- > > Key: HIVE-13559 > URL: https://issues.apache.org/jira/browse/HIVE-13559 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13559.1.patch > > > Pass exception to failure hooks so that they know more about the failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13559) Pass exception to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13559: --- Attachment: HIVE-13559.1.patch > Pass exception to failure hooks > --- > > Key: HIVE-13559 > URL: https://issues.apache.org/jira/browse/HIVE-13559 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13559.1.patch > > > Pass exception to failure hooks so that they know more about the failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13530) Hive on Spark throws Kryo exception in some cases
[
https://issues.apache.org/jira/browse/HIVE-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244988#comment-15244988
]
Jimmy Xiang commented on HIVE-13530:
This patch should fix this issue. However, I was wondering if we really need to
serialize the perflogger object? If not, can we just fix class
SparkHashTableSinkOperator not to use perflogger as a member field?
> Hive on Spark throws Kryo exception in some cases
> -
>
> Key: HIVE-13530
> URL: https://issues.apache.org/jira/browse/HIVE-13530
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-13530.patch
>
>
> After recent changes, Hive on Spark throws KryoException:
> {noformat}
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: ERROR : Failed to execute spark
> task, with exception 'java.lang.Exception(Failed to submit Spark work, please
> retry later)'
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: java.lang.Exception: Failed to
> submit Spark work, please retry later
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.execute(RemoteHiveSparkClient.java:174)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:71)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:103)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1769)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1526)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1305)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1114)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1107)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.security.AccessController.doPrivileged(Native Method)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.lang.Thread.run(Thread.java:745)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Caused by:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.util.ConcurrentModificationException
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Serialization trace:
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classes
> (sun.misc.Launcher$AppClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classloader
> (java.security.ProtectionDomain)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: context
> (java.security.AccessControlContext)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: acc (java.net.URLClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classLoader
> (org.apache.hadoop.hive.conf.HiveConf)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: conf
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: metrics
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: openScopes
> (org.apache.hadoop.hive.ql.log.PerfLogger)
> 16/04/14 21:53:24 INFO
[jira] [Updated] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13501: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Szehon for the review. > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13501.1.patch, HIVE-13501.2.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13501: --- Attachment: HIVE-13501.2.patch Rebased patch 1 to master latest. > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13501.1.patch, HIVE-13501.2.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239859#comment-15239859 ] Jimmy Xiang commented on HIVE-13501: Thanks for the review. For now, I have not seen such an exception yet :) Just want to make sure all scenarios are covered. > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13501.1.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13501: --- Attachment: HIVE-13501.1.patch > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13501.1.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13501: --- Status: Patch Available (was: Open) > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13501.1.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13430) Pass error message to failure hook
[ https://issues.apache.org/jira/browse/HIVE-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13430: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) These test failures are not related. Integrated into trunk. Thanks Szehon for the review. > Pass error message to failure hook > -- > > Key: HIVE-13430 > URL: https://issues.apache.org/jira/browse/HIVE-13430 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13430.1.patch > > > Currently, the failure hook just knows the query failed. But it has no clue > what the error is. It is better to pass the error message to the hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13430) Pass error message to failure hook
[ https://issues.apache.org/jira/browse/HIVE-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13430: --- Status: Patch Available (was: Open) > Pass error message to failure hook > -- > > Key: HIVE-13430 > URL: https://issues.apache.org/jira/browse/HIVE-13430 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13430.1.patch > > > Currently, the failure hook just knows the query failed. But it has no clue > what the error is. It is better to pass the error message to the hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13430) Pass error message to failure hook
[ https://issues.apache.org/jira/browse/HIVE-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13430: --- Attachment: HIVE-13430.1.patch > Pass error message to failure hook > -- > > Key: HIVE-13430 > URL: https://issues.apache.org/jira/browse/HIVE-13430 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13430.1.patch > > > Currently, the failure hook just knows the query failed. But it has no clue > what the error is. It is better to pass the error message to the hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12619: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Sergio for the review. Thanks Mohammad for the initial patch. > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch, HIVE-12619.6.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216754#comment-15216754 ] Jimmy Xiang commented on HIVE-12619: [~spena], could you take a look patch v6 when you get a chance? Thanks. > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch, HIVE-12619.6.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205252#comment-15205252 ] Jimmy Xiang commented on HIVE-12619: Patch v6 is uploaded to RB: https://reviews.apache.org/r/45128/ > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch, HIVE-12619.6.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12619: --- Status: Patch Available (was: Open) > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch, HIVE-12619.6.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12619: --- Attachment: HIVE-12619.6.patch > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch, HIVE-12619.6.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204960#comment-15204960 ] Jimmy Xiang commented on HIVE-12619: Sure. I will upload the next patch to RB. > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13141) Hive on Spark over HBase should accept parameters starting with "zookeeper.znode"
[ https://issues.apache.org/jira/browse/HIVE-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13141: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Thanks Nemon for the patch. Integrated into trunk. > Hive on Spark over HBase should accept parameters starting with > "zookeeper.znode" > - > > Key: HIVE-13141 > URL: https://issues.apache.org/jira/browse/HIVE-13141 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.0, 2.0.0 >Reporter: Nemon Lou >Assignee: Nemon Lou >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13141.patch > > > HBase related paramters has been added by HIVE-12708. > Following the same way,parameters starting with "zookeeper.znode" should be > add too,which are also HBase related paramters . > Refering to http://blog.cloudera.com/blog/2013/10/what-are-hbase-znodes/ > I have seen a failure with Hive on Spark over HBase due to customize > zookeeper.znode.parent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200356#comment-15200356 ] Jimmy Xiang commented on HIVE-12619: [~spena], v3 doesn't work with deeper levels. So I enhanced v2 a little, modified the testcase, and got v4. > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12619: --- Attachment: HIVE-12619.5.patch > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch, > HIVE-12619.4.patch, HIVE-12619.5.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
