[jira] [Commented] (HIVE-25107) Thread classpath logging should be optional/DEBUG
[
https://issues.apache.org/jira/browse/HIVE-25107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343342#comment-17343342
]
Zoltan Matyus commented on HIVE-25107:
--
My intention back then was to keep roughly the same logging as before. Now,
having reviewed my change, I see that in one instance I effectively changed the
level from DEBUG to INFO (for tez/RecordProcessor.init). In all other cases,
{{tryLoggingClassPaths}} was added to replace two LOG.info calls (or two
l4j.info, etc).
So, at least for RecordProcessor.init, we definitely could change the level
back to DEBUG. But I think we might as well change the level to DEBUG in all
other cases as well, we don't need to stick to the previous behavior
necessarily. Let me know if you expect me to contribute to this.
> Thread classpath logging should be optional/DEBUG
> -
>
> Key: HIVE-25107
> URL: https://issues.apache.org/jira/browse/HIVE-25107
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> This is since HIVE-21584
> I have a *72M* llap executor log file, then I grepped for only "thread class
> path", piped into a separate file and a result was a *22M* file...1/3-1/4 of
> the file was classpath info which is not usable for most of the time. This
> overwhelming amount of classpath info is not needed, assuming that classpath
> issues are reproducible with more or less effort, user should be responsible
> for turning on this expensive logging on demand. Not to mention performance
> implications which cannot be ignored beyond a certain amount of log messages.
> https://github.com/apache/hive/commit/a234475faa2cab2606f2a74eb9ca071f006998e2#diff-44b2ff3a3c4a6cfcaed0fcb40b74031844f8586e40a6f8261637e5ebcd558b73R4577
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24734) Sanity check in HiveSplitGenerator available slot calculation
[
https://issues.apache.org/jira/browse/HIVE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284595#comment-17284595
]
Zoltan Matyus commented on HIVE-24734:
--
I'm not about to investigate; since this happened on local dev setup.
You're right, we shouldn't auto-correct, but this definitely deserves an
exception: In my case, one of the values was correct, and only the other was
-1. However, I guess it's not impossible for both values to be the default /
unset -1; in which case we would end up with the hidden error. So I'll add
separate checking for the resource values and throw an exception (when I get
around to it).
> Sanity check in HiveSplitGenerator available slot calculation
> -
>
> Key: HIVE-24734
> URL: https://issues.apache.org/jira/browse/HIVE-24734
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Priority: Minor
>
> HiveSplitGenerator calculates the number of available slots from available
> memory like this:
> {code:java}
> if (getContext() != null) {
> totalResource = getContext().getTotalAvailableResource().getMemory();
> taskResource = getContext().getVertexTaskResource().getMemory();
> availableSlots = totalResource / taskResource;
> }
> {code}
> I had a scenario where the total memory was calculated correctly, but the
> task memory returned -1. This led to error like these:
> {noformat}
> tez.HiveSplitGenerator: Number of input splits: 1. -3641 available slots, 1.7
> waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat
> Estimated number of tasks: -6189 for bucket 1
> java.lang.IllegalArgumentException: Illegal Capacity: -6189
> {noformat}
> Admittedly, this happened during development, and hopefully will not occur on
> a properly configured cluster. (Although I'm not sure what the issue was on
> my setup, possibly XMX set higher than physical memory.)
> In any case, it feels like setting availableSlots < 1 will never lead to
> desired behavior, so in such cases we could emit a warning and correct the
> value to 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24734) Sanity check in HiveSplitGenerator available slot calculation
[
https://issues.apache.org/jira/browse/HIVE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282660#comment-17282660
]
Zoltan Matyus commented on HIVE-24734:
--
[~abstractdog]: could you comment on this? This is probably an edge case, but I
see no harm in making the change...
> Sanity check in HiveSplitGenerator available slot calculation
> -
>
> Key: HIVE-24734
> URL: https://issues.apache.org/jira/browse/HIVE-24734
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Priority: Minor
>
> HiveSplitGenerator calculates the number of available slots from available
> memory like this:
> {code:java}
> if (getContext() != null) {
> totalResource = getContext().getTotalAvailableResource().getMemory();
> taskResource = getContext().getVertexTaskResource().getMemory();
> availableSlots = totalResource / taskResource;
> }
> {code}
> I had a scenario where the total memory was calculated correctly, but the
> task memory returned -1. This led to error like these:
> {noformat}
> tez.HiveSplitGenerator: Number of input splits: 1. -3641 available slots, 1.7
> waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat
> Estimated number of tasks: -6189 for bucket 1
> java.lang.IllegalArgumentException: Illegal Capacity: -6189
> {noformat}
> Admittedly, this happened during development, and hopefully will not occur on
> a properly configured cluster. (Although I'm not sure what the issue was on
> my setup, possibly XMX set higher than physical memory.)
> In any case, it feels like setting availableSlots < 1 will never lead to
> desired behavior, so in such cases we could emit a warning and correct the
> value to 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24734) Sanity check in HiveSplitGenerator available slot calculation
[
https://issues.apache.org/jira/browse/HIVE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278679#comment-17278679
]
Zoltan Matyus commented on HIVE-24734:
--
My proposal would be this:
{code:java}
if (getContext() != null) {
totalResource = getContext().getTotalAvailableResource().getMemory();
taskResource = getContext().getVertexTaskResource().getMemory();
availableSlots = totalResource / taskResource;
LOG.info(
"Using availableSlots: {} calculated from (total memory: {} / task
memory: {})",
availableSlots, totalResource, taskResource);
if(availableSlots < 1) {
LOG.warn("Correcting availableSlots to 1 (from {}).", availableSlots);
availableSlots = 1;
}
}
{code}
I'm not opening a PR yet, I'm not a Tez expert and I'd like some opinions
before starting the precommit machinery...
> Sanity check in HiveSplitGenerator available slot calculation
> -
>
> Key: HIVE-24734
> URL: https://issues.apache.org/jira/browse/HIVE-24734
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Priority: Minor
>
> HiveSplitGenerator calculates the number of available slots from available
> memory like this:
> {code:java}
> if (getContext() != null) {
> totalResource = getContext().getTotalAvailableResource().getMemory();
> taskResource = getContext().getVertexTaskResource().getMemory();
> availableSlots = totalResource / taskResource;
> }
> {code}
> I had a scenario where the total memory was calculated correctly, but the
> task memory returned -1. This led to error like these:
> {noformat}
> tez.HiveSplitGenerator: Number of input splits: 1. -3641 available slots, 1.7
> waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat
> Estimated number of tasks: -6189 for bucket 1
> java.lang.IllegalArgumentException: Illegal Capacity: -6189
> {noformat}
> Admittedly, this happened during development, and hopefully will not occur on
> a properly configured cluster. (Although I'm not sure what the issue was on
> my setup, possibly XMX set higher than physical memory.)
> In any case, it feels like setting availableSlots < 1 will never lead to
> desired behavior, so in such cases we could emit a warning and correct the
> value to 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24636) Memory leak due to stacking UDFClassLoader in Apache Commons LogFactory
[ https://issues.apache.org/jira/browse/HIVE-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus resolved HIVE-24636. -- Fix Version/s: 3.2.0 3.1.3 Assignee: Zoltan Matyus Resolution: Fixed This is fixed on 3.1.x and 3.x branches. Not applicable on master. > Memory leak due to stacking UDFClassLoader in Apache Commons LogFactory > --- > > Key: HIVE-24636 > URL: https://issues.apache.org/jira/browse/HIVE-24636 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: dohongdayi >Assignee: Zoltan Matyus >Priority: Major > Labels: pull-request-available > Fix For: 3.1.3, 3.2.0 > > Attachments: HIVE-24636.1.patch.txt > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Much the same as [HIVE-7563|https://issues.apache.org/jira/browse/HIVE-7563], > after ClassLoader is closed in JavaUtils, it should be released by Apache > Commons LogFactory, or the ClassLoader can't be Garbage Collected, which > leads to memory leak, exactly our PROD met. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24691) Ban commons-logging (again)
[ https://issues.apache.org/jira/browse/HIVE-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus resolved HIVE-24691. -- Resolution: Fixed > Ban commons-logging (again) > --- > > Key: HIVE-24691 > URL: https://issues.apache.org/jira/browse/HIVE-24691 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Zoltan Matyus >Assignee: Zoltan Matyus >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The usage of commons-logging has been completely removed once from Hive in > HIVE-20019. However, new usage has been added since, despite attempts to ban > this (bannedDependencies). I'm removing all usage again, and add another way > to ban using it (restrictImports). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24636) Memory leak due to stacking UDFClassLoader in Apache Commons LogFactory
[ https://issues.apache.org/jira/browse/HIVE-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272872#comment-17272872 ] Zoltan Matyus commented on HIVE-24636: -- Status: I mistakenly created a fix and a PR to _master_. Later I realized that LogFactory should not be used on _master_ at all (see HIVE-24691). I abandoned [PR #1910|https://github.com/apache/hive/pull/1910] and started the process in a new [PR #1917|https://github.com/apache/hive/pull/1917] for _branch-3_. I'm not sure how to get the change to _branch-3.1_, probably a new cherry-pick and new PR (after the change is merged to _branch-3_). > Memory leak due to stacking UDFClassLoader in Apache Commons LogFactory > --- > > Key: HIVE-24636 > URL: https://issues.apache.org/jira/browse/HIVE-24636 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: dohongdayi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24636.1.patch.txt > > Time Spent: 40m > Remaining Estimate: 0h > > Much the same as [HIVE-7563|https://issues.apache.org/jira/browse/HIVE-7563], > after ClassLoader is closed in JavaUtils, it should be released by Apache > Commons LogFactory, or the ClassLoader can't be Garbage Collected, which > leads to memory leak, exactly our PROD met. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24691) Ban commons-logging (again)
[ https://issues.apache.org/jira/browse/HIVE-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272842#comment-17272842 ] Zoltan Matyus commented on HIVE-24691: -- Note: I found that commons-logging was still used on master when I was trying to fix HIVE-24636. > Ban commons-logging (again) > --- > > Key: HIVE-24691 > URL: https://issues.apache.org/jira/browse/HIVE-24691 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Zoltan Matyus >Assignee: Zoltan Matyus >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The usage of commons-logging has been completely removed once from Hive in > HIVE-20019. However, new usage has been added since, despite attempts to ban > this (bannedDependencies). I'm removing all usage again, and add another way > to ban using it (restrictImports). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24691) Ban commons-logging (again)
[ https://issues.apache.org/jira/browse/HIVE-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus reassigned HIVE-24691: > Ban commons-logging (again) > --- > > Key: HIVE-24691 > URL: https://issues.apache.org/jira/browse/HIVE-24691 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Zoltan Matyus >Assignee: Zoltan Matyus >Priority: Minor > > The usage of commons-logging has been completely removed once from Hive in > HIVE-20019. However, new usage has been added since, despite attempts to ban > this (bannedDependencies). I'm removing all usage again, and add another way > to ban using it (restrictImports). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24636) Memory leak due to stacking UDFClassLoader in Apache Commons LogFactory
[ https://issues.apache.org/jira/browse/HIVE-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271554#comment-17271554 ] Zoltan Matyus commented on HIVE-24636: -- Hi [~dohongdayi], appreciate the patch, but I think the project now uses github pull requests, not patch files. In any case, this is not resolved fixed, since it has not been reviewed and merged by a committer. Do you need help pushing this through the process? > Memory leak due to stacking UDFClassLoader in Apache Commons LogFactory > --- > > Key: HIVE-24636 > URL: https://issues.apache.org/jira/browse/HIVE-24636 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: dohongdayi >Assignee: dohongdayi >Priority: Major > Attachments: HIVE-24636.1.patch.txt > > > Much the same as [HIVE-7563|https://issues.apache.org/jira/browse/HIVE-7563], > after ClassLoader is closed in JavaUtils, it should be released by Apache > Commons LogFactory, or the ClassLoader can't be Garbage Collected, which > leads to memory leak, exactly our PROD met. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-2301) Throw error when attempting to create a column with the same name as a partition column
[
https://issues.apache.org/jira/browse/HIVE-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus resolved HIVE-2301.
-
Resolution: Done
> Throw error when attempting to create a column with the same name as a
> partition column
> ---
>
> Key: HIVE-2301
> URL: https://issues.apache.org/jira/browse/HIVE-2301
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.8.0
>Reporter: Paul Yang
>Assignee: Zoltan Matyus
>Priority: Minor
>
> If an alter table is run to rename a column to the same name as a partition
> column, the alter will succeed. However, subsequent operations on that table
> will fail.
> {code}
> hive> create table tmp_pyang_test (key string) partitioned by (ds string);
> OK
> Time taken: 4.773 seconds
> hive> alter table tmp_pyang_test replace columns (ds string);
> OK
> Time taken: 1.254 seconds
> hive> describe tmp_pyang_test;
> FAILED: Error in metadata: Partition column name ds conflicts with table
> columns.
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> hive>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-2301) Throw error when attempting to create a column with the same name as a partition column
[
https://issues.apache.org/jira/browse/HIVE-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-2301:
---
Assignee: Zoltan Matyus
> Throw error when attempting to create a column with the same name as a
> partition column
> ---
>
> Key: HIVE-2301
> URL: https://issues.apache.org/jira/browse/HIVE-2301
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.8.0
>Reporter: Paul Yang
>Assignee: Zoltan Matyus
>Priority: Minor
>
> If an alter table is run to rename a column to the same name as a partition
> column, the alter will succeed. However, subsequent operations on that table
> will fail.
> {code}
> hive> create table tmp_pyang_test (key string) partitioned by (ds string);
> OK
> Time taken: 4.773 seconds
> hive> alter table tmp_pyang_test replace columns (ds string);
> OK
> Time taken: 1.254 seconds
> hive> describe tmp_pyang_test;
> FAILED: Error in metadata: Partition column name ds conflicts with table
> columns.
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> hive>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-2301) Throw error when attempting to create a column with the same name as a partition column
[
https://issues.apache.org/jira/browse/HIVE-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-2301 started by Zoltan Matyus.
---
> Throw error when attempting to create a column with the same name as a
> partition column
> ---
>
> Key: HIVE-2301
> URL: https://issues.apache.org/jira/browse/HIVE-2301
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.8.0
>Reporter: Paul Yang
>Assignee: Zoltan Matyus
>Priority: Minor
>
> If an alter table is run to rename a column to the same name as a partition
> column, the alter will succeed. However, subsequent operations on that table
> will fail.
> {code}
> hive> create table tmp_pyang_test (key string) partitioned by (ds string);
> OK
> Time taken: 4.773 seconds
> hive> alter table tmp_pyang_test replace columns (ds string);
> OK
> Time taken: 1.254 seconds
> hive> describe tmp_pyang_test;
> FAILED: Error in metadata: Partition column name ds conflicts with table
> columns.
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> hive>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24213) Incorrect exception in the Merge MapJoinTask into its child MapRedTask optimizer
[
https://issues.apache.org/jira/browse/HIVE-24213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-24213:
-
Attachment: (was: HIVE-24213.patch)
> Incorrect exception in the Merge MapJoinTask into its child MapRedTask
> optimizer
>
>
> Key: HIVE-24213
> URL: https://issues.apache.org/jira/browse/HIVE-24213
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
>
> The {{CommonJoinTaskDispatcher#mergeMapJoinTaskIntoItsChildMapRedTask}}
> method throws a {{SemanticException}} if the number of {{FileSinkOperator}}s
> it finds is not exactly 1. The exception is valid if zero operators are
> found, but there can be valid use cases where multiple FileSinkOperators
> exist.
> Example: the MapJoin and it child are used in a common table expression,
> which is used for multiple inserts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24213) Incorrect exception in the Merge MapJoinTask into its child MapRedTask optimizer
[
https://issues.apache.org/jira/browse/HIVE-24213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-24213:
-
Status: Patch Available (was: In Progress)
> Incorrect exception in the Merge MapJoinTask into its child MapRedTask
> optimizer
>
>
> Key: HIVE-24213
> URL: https://issues.apache.org/jira/browse/HIVE-24213
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-24213.patch
>
>
> The {{CommonJoinTaskDispatcher#mergeMapJoinTaskIntoItsChildMapRedTask}}
> method throws a {{SemanticException}} if the number of {{FileSinkOperator}}s
> it finds is not exactly 1. The exception is valid if zero operators are
> found, but there can be valid use cases where multiple FileSinkOperators
> exist.
> Example: the MapJoin and it child are used in a common table expression,
> which is used for multiple inserts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-24213) Incorrect exception in the Merge MapJoinTask into its child MapRedTask optimizer
[
https://issues.apache.org/jira/browse/HIVE-24213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-24213 started by Zoltan Matyus.
> Incorrect exception in the Merge MapJoinTask into its child MapRedTask
> optimizer
>
>
> Key: HIVE-24213
> URL: https://issues.apache.org/jira/browse/HIVE-24213
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-24213.patch
>
>
> The {{CommonJoinTaskDispatcher#mergeMapJoinTaskIntoItsChildMapRedTask}}
> method throws a {{SemanticException}} if the number of {{FileSinkOperator}}s
> it finds is not exactly 1. The exception is valid if zero operators are
> found, but there can be valid use cases where multiple FileSinkOperators
> exist.
> Example: the MapJoin and it child are used in a common table expression,
> which is used for multiple inserts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24213) Incorrect exception in the Merge MapJoinTask into its child MapRedTask optimizer
[
https://issues.apache.org/jira/browse/HIVE-24213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-24213:
-
Attachment: HIVE-24213.patch
> Incorrect exception in the Merge MapJoinTask into its child MapRedTask
> optimizer
>
>
> Key: HIVE-24213
> URL: https://issues.apache.org/jira/browse/HIVE-24213
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-24213.patch
>
>
> The {{CommonJoinTaskDispatcher#mergeMapJoinTaskIntoItsChildMapRedTask}}
> method throws a {{SemanticException}} if the number of {{FileSinkOperator}}s
> it finds is not exactly 1. The exception is valid if zero operators are
> found, but there can be valid use cases where multiple FileSinkOperators
> exist.
> Example: the MapJoin and it child are used in a common table expression,
> which is used for multiple inserts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24213) Incorrect exception in the Merge MapJoinTask into its child MapRedTask optimizer
[
https://issues.apache.org/jira/browse/HIVE-24213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-24213:
> Incorrect exception in the Merge MapJoinTask into its child MapRedTask
> optimizer
>
>
> Key: HIVE-24213
> URL: https://issues.apache.org/jira/browse/HIVE-24213
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
>
> The {{CommonJoinTaskDispatcher#mergeMapJoinTaskIntoItsChildMapRedTask}}
> method throws a {{SemanticException}} if the number of {{FileSinkOperator}}s
> it finds is not exactly 1. The exception is valid if zero operators are
> found, but there can be valid use cases where multiple FileSinkOperators
> exist.
> Example: the MapJoin and it child are used in a common table expression,
> which is used for multiple inserts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024387#comment-17024387
]
Zoltan Matyus commented on HIVE-22767:
--
bq. I think instead of allocating '|' to a custom role in the test [...] the
following could have been an alternative [...]
Right. The intention was to have tests that are (1) one-liners and (2)
readable. Clearly, neither condition was met :)
bq. further more...the helper method in the test writes out information to the
terminal; instead of making it part of the exception
I didn't mean to push that, thanks for spotting it. Anyway, I switched from
{{assertArrayEquals}} to {{assertEquals(List)}} this is a bit more readable.
This actually made {{invokeGetCmdList}} unnecessary.
So I rewrite the testing part, pushed patch 3. *No code changes*, and no
substantial changes to tests, just refactor... [~kgyrtkirk], could you take
another look when the tests are done? Thanks
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch, HIVE-22767.02.patch,
> HIVE-22767.03.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22767:
-
Attachment: HIVE-22767.03.patch
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch, HIVE-22767.02.patch,
> HIVE-22767.03.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22767:
-
Attachment: HIVE-22767.02.patch
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch, HIVE-22767.02.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22767:
-
Status: Patch Available (was: Open)
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22767:
-
Attachment: HIVE-22767.01.patch
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-22767:
Assignee: Zoltan Matyus
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22208:
-
Attachment: HIVE-22208.02.patch
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Zoltan Matyus
>Priority: Critical
> Attachments: HIVE-22208.01.patch, HIVE-22208.02.patch
>
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22236:
-
Attachment: HIVE-22236.03.patch
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-22236.01.patch, HIVE-22236.02.patch,
> HIVE-22236.03.patch, HIVE-22236.q
>
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938384#comment-16938384
]
Zoltan Matyus commented on HIVE-22208:
--
[~jcamachorodriguez] I though I was pretty clever and solved this by calling
{{SemanticAnalyzer.getQueryStringFromAst}} instead of
{{TokenRewriteStream.toString}}. It seemed to be exactly for this purpose. But
I broke masking and accidentally committed the results (running the tests with
overwrite).
Could you take a look at the patch (the code change only, disregard the
modified q.out files) and give any hints? It seems it's not the parsing but the
rewrite that needs to be fixed, right? (Based on this:
bq. // The Tokens have no distinction between Identifiers and QuotedIdentifiers.
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Zoltan Matyus
>Priority: Critical
> Attachments: HIVE-22208.01.patch
>
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22208:
-
Status: Patch Available (was: Open)
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Zoltan Matyus
>Priority: Critical
> Attachments: HIVE-22208.01.patch
>
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22208:
-
Attachment: HIVE-22208.01.patch
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Zoltan Matyus
>Priority: Critical
> Attachments: HIVE-22208.01.patch
>
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-22208:
Assignee: Zoltan Matyus (was: Jesus Camacho Rodriguez)
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Zoltan Matyus
>Priority: Critical
> Attachments: HIVE-22208.01.patch
>
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22236:
-
Attachment: HIVE-22236.02.patch
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-22236.01.patch, HIVE-22236.02.patch, HIVE-22236.q
>
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22236:
-
Status: Patch Available (was: Open)
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-22236.01.patch, HIVE-22236.q
>
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22236:
-
Attachment: HIVE-22236.01.patch
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-22236.01.patch, HIVE-22236.q
>
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936643#comment-16936643
]
Zoltan Matyus commented on HIVE-22236:
--
Attached minimum script to reproduce.
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-22236.q
>
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22236:
-
Attachment: HIVE-22236.q
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
> Attachments: HIVE-22236.q
>
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22236) Fail to create View selecting View containing NOT IN subquery
[
https://issues.apache.org/jira/browse/HIVE-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-22236:
> Fail to create View selecting View containing NOT IN subquery
> -
>
> Key: HIVE-22236
> URL: https://issues.apache.org/jira/browse/HIVE-22236
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Minor
>
> * Given a complicated view with a select statement that has subquery
> containing "{{NOT IN}}"
> * Hive fails to create a simple view as {{SELECT * FROM complicated_view}}
> * (with CBO disabled).
> The unparse replacements of the complicated view will be applied to the text
> of the simple view, resulting in {{IllegalArgumentException: replace: range
> invalid}} exceptions from {{org.antlr.runtime.TokenRewriteStream.replace}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus updated HIVE-21796: - Attachment: HIVE-21796.5.patch > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.4.patch, HIVE-21796.5.patch, HIVE-21796.patch, > jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus updated HIVE-21796: - Attachment: HIVE-21796.4.patch > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.4.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus updated HIVE-21796: - Attachment: HIVE-21796.3.patch > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857503#comment-16857503 ] Zoltan Matyus commented on HIVE-21796: -- bq. ERROR: -1 due to 16 failed/errored test(s), 16082 tests executed Failed tests are unrelated, retrying with same content. > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus reassigned HIVE-21796: Assignee: Zoltan Matyus (was: Csaba Ringhofer) > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.10.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch, HIVE-21584.05.patch,
> HIVE-21584.06.patch, HIVE-21584.07.patch, HIVE-21584.08.patch,
> HIVE-21584.09.patch, HIVE-21584.10.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.09.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch, HIVE-21584.05.patch,
> HIVE-21584.06.patch, HIVE-21584.07.patch, HIVE-21584.08.patch,
> HIVE-21584.09.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.08.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch, HIVE-21584.05.patch,
> HIVE-21584.06.patch, HIVE-21584.07.patch, HIVE-21584.08.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.07.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch, HIVE-21584.05.patch,
> HIVE-21584.06.patch, HIVE-21584.07.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.06.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch, HIVE-21584.05.patch,
> HIVE-21584.06.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.05.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch, HIVE-21584.05.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818033#comment-16818033
]
Zoltan Matyus commented on HIVE-21584:
--
Patch 04 attempts to fix [review
comments|https://reviews.apache.org/r/70453/#comment300813] from [~szita].
Also, removed some workaround from {{org.apache.hadoop.hive.common.JavaUtils}},
that was introduced by HIVE-3969 to solve Java 1.6-specific problems
({{URLClassLoader}} doesn't implement {{Closeable}}) with _"undocumented,
unpublished, private"_ methods (i.e. {{sun.misc.ClassLoaderUtil}}). This is not
relevant anymore, as Hive requires Java 1.7 at least – also super unlikely to
work on any recent Java versions.
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.04.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch, HIVE-21584.04.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.03.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch,
> HIVE-21584.03.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814274#comment-16814274
]
Zoltan Matyus commented on HIVE-21584:
--
bq. org.apache.hadoop.hive.ql.exec.Utilities.createUDFClassLoader(ClassLoader,
String[]) creates a org.apache.hadoop.hive.ql.exec.UDFClassLoader classloader,
which should be performed within a doPrivileged block At
Utilities.java:classloader, which should be performed within a doPrivileged
block At Utilities.java:[line 2126]
Creating class loader is not new functionality, but now that I modified the
code, FindBugs complains. This could be solved fairly easily, by hiding the
{{AccessController.doPrivileged}} call in the CreateUDFClassLoader method.
However, Oracle [strongly
recommends|https://docs.oracle.com/javase/8/docs/technotes/guides/security/doprivileged.html]
not to do this:
bq. Do not be tempted to write a utility class that itself calls doPrivileged
as that could lead to security holes. You can write utility classes for
PrivilegedAction classes though, as shown in the preceding example.
I followed this approach, which meant a small change at each caller's location.
Also, I merged the addToClassPath and the createNewClassLoader functionality
into an AddToClassPath action with a forceNewClassLoader flag...
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.02.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch, HIVE-21584.02.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Status: Patch Available (was: Open)
First patch, possibly incomplete / incorrect.
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-21584:
-
Attachment: HIVE-21584.01.patch
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21584.01.patch
>
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21584) Java 11 preparation: system class loader is not URLClassLoader
[
https://issues.apache.org/jira/browse/HIVE-21584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-21584:
> Java 11 preparation: system class loader is not URLClassLoader
> --
>
> Key: HIVE-21584
> URL: https://issues.apache.org/jira/browse/HIVE-21584
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
>
> Currently, Hive assumes that the system class loader is instance of
> {{URLClassLoader}}. In Java 11 this is not the case. There are a few
> (unresolved) JIRAs about specific occurrences of {{URLClassLoader}} (e.g.
> [HIVE-21237|https://issues.apache.org/jira/browse/HIVE-21237],
> [HIVE-17909|https://issues.apache.org/jira/browse/HIVE-17909]), but no
> _"remove all occurrences"_. Also I couldn't find umbrella "Java 11 upgrade"
> JIRA.
> This ticket is to remove all unconditional casts of any random class loader
> to {{URLClassLoader}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
