[jira] [Commented] (HIVE-20295) Remove !isNumber check after failed constant interpretation
[
https://issues.apache.org/jira/browse/HIVE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760552#comment-16760552
]
Hive QA commented on HIVE-20295:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
27s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} ql generated 0 new + 2303 unchanged - 4 fixed = 2303
total (was 2307) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15940/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15940/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15940/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove !isNumber check after failed constant interpretation
> ---
>
> Key: HIVE-20295
> URL: https://issues.apache.org/jira/browse/HIVE-20295
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-20295.01.patch, HIVE-20295.02.patch,
> HIVE-20295.03.patch, HIVE-20295.04.patch, HIVE-20295.05.patch,
> HIVE-20295.06.patch, HIVE-20295.07.patch, HIVE-20295.08.patch,
> HIVE-20295.09.patch
>
>
> During constant interpretation; if the number can't be parsed - it might be
> possible that the comparsion is out of range for the type in question - in
> which case it could be removed.
> https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java#L1163
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.22.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21214 started by Deepak Jaiswal.
-
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal reassigned HIVE-21214:
-
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20295) Remove !isNumber check after failed constant interpretation
[
https://issues.apache.org/jira/browse/HIVE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760590#comment-16760590
]
Hive QA commented on HIVE-20295:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957595/HIVE-20295.09.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15771 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15940/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15940/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15940/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957595 - PreCommit-HIVE-Build
> Remove !isNumber check after failed constant interpretation
> ---
>
> Key: HIVE-20295
> URL: https://issues.apache.org/jira/browse/HIVE-20295
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-20295.01.patch, HIVE-20295.02.patch,
> HIVE-20295.03.patch, HIVE-20295.04.patch, HIVE-20295.05.patch,
> HIVE-20295.06.patch, HIVE-20295.07.patch, HIVE-20295.08.patch,
> HIVE-20295.09.patch
>
>
> During constant interpretation; if the number can't be parsed - it might be
> possible that the comparsion is out of range for the type in question - in
> which case it could be removed.
> https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java#L1163
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20295) Remove !isNumber check after failed constant interpretation
[ https://issues.apache.org/jira/browse/HIVE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760595#comment-16760595 ] Zoltan Haindrich commented on HIVE-20295: - +1 > Remove !isNumber check after failed constant interpretation > --- > > Key: HIVE-20295 > URL: https://issues.apache.org/jira/browse/HIVE-20295 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-20295.01.patch, HIVE-20295.02.patch, > HIVE-20295.03.patch, HIVE-20295.04.patch, HIVE-20295.05.patch, > HIVE-20295.06.patch, HIVE-20295.07.patch, HIVE-20295.08.patch, > HIVE-20295.09.patch > > > During constant interpretation; if the number can't be parsed - it might be > possible that the comparsion is out of range for the type in question - in > which case it could be removed. > https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java#L1163 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[ https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller reassigned HIVE-21207: -- Assignee: Ivan Suller (was: Oleksiy Sayankin) > Use 0.12.0 libthrift version in Hive > > > Key: HIVE-21207 > URL: https://issues.apache.org/jira/browse/HIVE-21207 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21207.1.patch > > > Use 0.12.0 libthrift version in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[ https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760607#comment-16760607 ] Ivan Suller commented on HIVE-21207: Hi [~osayankin] This is a duplicate of HIVE-21000 - I'm already working on that ticket. > Use 0.12.0 libthrift version in Hive > > > Key: HIVE-21207 > URL: https://issues.apache.org/jira/browse/HIVE-21207 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21207.1.patch > > > Use 0.12.0 libthrift version in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Attachment: HIVE.685.08.PATCH > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.08.PATCH, > HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Status: In Progress (was: Patch Available) > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.08.PATCH, > HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Status: Patch Available (was: In Progress) > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.08.PATCH, > HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21199) Replace all occurences of new Byte with Byte.valueOf
[
https://issues.apache.org/jira/browse/HIVE-21199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760618#comment-16760618
]
Hive QA commented on HIVE-21199:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
24s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} beeline in master has 53 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} serde: The patch generated 0 new + 37 unchanged - 3
fixed = 37 total (was 40) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} ql: The patch generated 0 new + 141 unchanged - 13
fixed = 141 total (was 154) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch beeline passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} serde generated 0 new + 197 unchanged - 1 fixed =
197 total (was 198) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} ql generated 0 new + 2302 unchanged - 5 fixed = 2302
total (was 2307) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} beeline generated 0 new + 52 unchanged - 1 fixed =
52 total (was 53) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15941/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://1
[jira] [Assigned] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[ https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller reassigned HIVE-21207: -- Assignee: Oleksiy Sayankin (was: Ivan Suller) > Use 0.12.0 libthrift version in Hive > > > Key: HIVE-21207 > URL: https://issues.apache.org/jira/browse/HIVE-21207 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-21207.1.patch > > > Use 0.12.0 libthrift version in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[
https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760623#comment-16760623
]
Oleksiy Sayankin commented on HIVE-21207:
-
Hi [~isuller].
Hm... The header of HIVE-21000 tells me *Upgrade thrift to at least 0.10.0*,
but my target is {{0.12.0}}. Do you want to upgrade to {{0.12.0}} in scope of
HIVE-21000?
> Use 0.12.0 libthrift version in Hive
>
>
> Key: HIVE-21207
> URL: https://issues.apache.org/jira/browse/HIVE-21207
> Project: Hive
> Issue Type: Improvement
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-21207.1.patch
>
>
> Use 0.12.0 libthrift version in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[ https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760624#comment-16760624 ] Ivan Suller commented on HIVE-21207: Hi [~osayankin] Yes, that's the current version and I'm close to finish creating the patch with the generated code included. > Use 0.12.0 libthrift version in Hive > > > Key: HIVE-21207 > URL: https://issues.apache.org/jira/browse/HIVE-21207 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-21207.1.patch > > > Use 0.12.0 libthrift version in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[ https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760628#comment-16760628 ] Oleksiy Sayankin commented on HIVE-21207: - Ok, then I resolve this as duplicate of HIVE-21000. > Use 0.12.0 libthrift version in Hive > > > Key: HIVE-21207 > URL: https://issues.apache.org/jira/browse/HIVE-21207 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-21207.1.patch > > > Use 0.12.0 libthrift version in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21207) Use 0.12.0 libthrift version in Hive
[ https://issues.apache.org/jira/browse/HIVE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21207: Resolution: Duplicate Status: Resolved (was: Patch Available) > Use 0.12.0 libthrift version in Hive > > > Key: HIVE-21207 > URL: https://issues.apache.org/jira/browse/HIVE-21207 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-21207.1.patch > > > Use 0.12.0 libthrift version in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[
https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760637#comment-16760637
]
Peter Vary commented on HIVE-20849:
---
[~belugabehr]: Quick questions:
* Why have you not changed the following lines of log formatting?:
{code}
LOG.error("Reverse look up of column " + desc + " error!");
...
LOG.error("Cannot resolve " + desc.getTabAlias() + "." + desc.getColumn());
{code}
* Have you validated that we do not try to change the Maps where you used
{{Collections.emptyMap()}} to set empty ones, or we just rely on tests for the
validation of this?
Thanks,
Peter
> Review of ConstantPropagateProcFactory
> --
>
> Key: HIVE-20849
> URL: https://issues.apache.org/jira/browse/HIVE-20849
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch,
> HIVE-20849.2.patch, HIVE-20849.3.patch, HIVE-20849.4.patch,
> HIVE-20849.5.patch, HIVE-20849.6.patch
>
>
> I was looking at this class because it blasts a lot of useless (to an admin)
> information to the logs. Especially if the table has a lot of columns, I see
> big blocks of logging that are meaningless to me. I request that the logging
> is toned down to debug, and some other improvements to the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-685) add UDFquote
[
https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760648#comment-16760648
]
Peter Vary commented on HIVE-685:
-
[~rmsm...@gmail.com]: All of the failures are timeouts. So my guess would be
that we have a problem initializing the function. What I would do is check test
logs: [http://104.198.109.242/logs/PreCommit-HIVE-Build-15916/], and find some
of the failures
[http://104.198.109.242/logs/PreCommit-HIVE-Build-15916/failed/1-TestCliDriver-udf_upper.q-ctas_date.q-materialized_view_rewrite_part_2.q-and-27-more/],
and try to run them on my machine. Sometimes the qtests are conflicting, and
if the tests are not failing by themselves then I try to run the whole batch
for myself. You can check the jenkins console to find the tests run in any
given batch:
[https://builds.apache.org/job/PreCommit-HIVE-Build/15916/consoleFull]. For
TestCliDriver-udf_upper.q-ctas_date.q-materialized_view_rewrite_part_2.q-and-27-more
the following:
{code:java}
2019-02-04 12:09:35,721 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
QFileTestBatch [batchId=1, size=30, driver=TestCliDriver,
queryFilesProperty=qfile,
name=1-TestCliDriver-udf_upper.q-ctas_date.q-materialized_view_rewrite_part_2.q-and-27-more,
tests=[udf_upper.q, ctas_date.q, materialized_view_rewrite_part_2.q,
groupby_grouping_sets3.q, vector_decimal_5.q, vector_case_when_conversion.q,
bucket_map_join_spark4.q, timestamp_2.q,
schema_evol_orc_acid_table_update_llap_io.q, date_join1.q, constprog_type.q,
timestamp_ints_casts.q, udf_negative.q, orc_merge_diff_fs.q,
udf_substring_index.q, results_cache_lifetime.q, cross_prod_3.q, masking_12.q,
diff_part_input_formats.q, auto_join_without_localtask.q, join46.q,
ctas_uses_table_location.q, tez_bmj_schema_evolution.q, bucketmapjoin4.q,
udf_context_aware.q, authorization_non_id.q, mapjoin_test_outer.q,
vectorization_9.q, input15.q, udf_PI.q], isParallel=true,
moduleName=itests/qtest]
{code}
Thanks,
Peter
> add UDFquote
>
>
> Key: HIVE-685
> URL: https://issues.apache.org/jira/browse/HIVE-685
> Project: Hive
> Issue Type: New Feature
>Reporter: Namit Jain
>Assignee: Mani M
>Priority: Major
> Labels: todoc4.0, udf
> Fix For: 4.0.0
>
> Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH,
> HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.08.PATCH,
> HIVE.685.PATCH
>
>
> add UDFquote
> look at
> http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html
> for details
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760484#comment-16760484 ] Mani M edited comment on HIVE-685 at 2/5/19 9:21 AM: - Hi [~pvary] I have created the test case for genericudfquote function with the same format as used in lpad function, but 885 test cases are getting failed as given above, along with the other test cases. Can you throw some light to sort out this issue. was (Author: rmsm...@gmail.com): Hi @pvary I have created the test case for genericudfquote function with the same format as used in lpad function, but 885 test cases are getting failed as given above, along with the other test cases. Can you throw some light to sort out this issue. > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21199) Replace all occurences of new Byte with Byte.valueOf
[
https://issues.apache.org/jira/browse/HIVE-21199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760654#comment-16760654
]
Hive QA commented on HIVE-21199:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957594/HIVE-21199.04.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15714 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=155)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=86)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15941/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15941/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15941/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957594 - PreCommit-HIVE-Build
> Replace all occurences of new Byte with Byte.valueOf
>
>
> Key: HIVE-21199
> URL: https://issues.apache.org/jira/browse/HIVE-21199
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Trivial
> Attachments: HIVE-21199.01.patch, HIVE-21199.02.patch,
> HIVE-21199.03.patch, HIVE-21199.04.patch
>
>
> Creating Byte objects with new Byte(...) creates a new object, while
> Byte.valueOf(...) can be cached (and is actually cached in most if not all
> JVMs) thus reducing GC overhead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-21214:
--
Status: Patch Available (was: In Progress)
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Status: Open (was: Patch Available) > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.08.PATCH, > HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-685) add UDFquote
[
https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760666#comment-16760666
]
Mani M commented on HIVE-685:
-
HI [~pvary]
Thanks for your valuable info.
Checked the logs. In Test File at one place the {color:#FF}null{color}
value is getting passed, which throws an error. I had fixed this and pushing
now for the build. Let's see build status.
java.lang.NullPointerException at
org.apache.hadoop.io.Text.encode(Text.java:451) at
org.apache.hadoop.io.Text.set(Text.java:198) at
org.apache.hadoop.io.Text.(Text.java:88) at
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFQuote.runAndVerify(TestGenericUDFQuote.java:52)
at
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFQuote.testQuote({color:#FF}TestGenericUDFQuote.java:39{color})
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
junit.framework.TestCase.runTest(TestCase.java:176) at
junit.framework.TestCase.runBare(TestCase.java:141) at
junit.framework.TestResult$1.protect(TestResult.java:122) at
junit.framework.TestResult.runProtected(TestResult.java:142) at
junit.framework.TestResult.run(TestResult.java:125) at
junit.framework.TestCase.run(TestCase.java:129) at
junit.framework.TestSuite.runTest(TestSuite.java:255) at
junit.framework.TestSuite.run(TestSuite.java:250) a
> add UDFquote
>
>
> Key: HIVE-685
> URL: https://issues.apache.org/jira/browse/HIVE-685
> Project: Hive
> Issue Type: New Feature
>Reporter: Namit Jain
>Assignee: Mani M
>Priority: Major
> Labels: todoc4.0, udf
> Fix For: 4.0.0
>
> Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH,
> HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.08.PATCH,
> HIVE.685.PATCH
>
>
> add UDFquote
> look at
> http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html
> for details
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-21214:
--
Attachment: HIVE-21214.1.patch
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Attachment: (was: HIVE.685.08.PATCH) > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21000: Attachment: HIVE-21000.01.patch > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Attachment: HIVE.685.09.PATCH > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.09.PATCH, > HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Status: Patch Available (was: Open) > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH, > HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.09.PATCH, > HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760691#comment-16760691
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
28s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} ql: The patch generated 0 new + 16 unchanged - 2
fixed = 16 total (was 18) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15942/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15942/yetus/patch-asflicense-problems.txt
|
| modules | C: ql accumulo-handler hbase-handler . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15942/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760699#comment-16760699
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957598/HIVE-21001.22.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15729 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_csv]
(batchId=275)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=275)
org.apache.hadoop.hive.metastore.TestMarkPartitionRemote.testMarkingPartitionSet
(batchId=230)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15942/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15942/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15942/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957598 - PreCommit-HIVE-Build
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-685) add UDFquote
[
https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760746#comment-16760746
]
Hive QA commented on HIVE-685:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
36s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15943/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15943/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15943/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add UDFquote
>
>
> Key: HIVE-685
> URL: https://issues.apache.org/jira/browse/HIVE-685
> Project: Hive
> Issue Type: New Feature
>Reporter: Namit Jain
>Assignee: Mani M
>Priority: Major
> Labels: todoc4.0, udf
> Fix For: 4.0.0
>
> Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH,
> HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.09.PATCH,
> HIVE.685.PATCH
>
>
> add UDFquote
> look at
> http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html
> for details
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[
https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760819#comment-16760819
]
BELUGA BEHR commented on HIVE-20849:
[~pvary] Thanks for the review!
# Do you mean why did I not use parameters in the logging? First, I wanted to
change as little as possible. I know you prefer that. :) Second, using
parameters is the fastest way to *not* log [1], but it would be very unlikely
that someone would configure their logging to be CRITICAL level only, so in
practice, these messages will always be logged; there is no need to optimize.
# The {{Collections.emptyMap()}} is relying on unit tests for verification.
[1] https://www.slf4j.org/faq.html#logging_performance
> Review of ConstantPropagateProcFactory
> --
>
> Key: HIVE-20849
> URL: https://issues.apache.org/jira/browse/HIVE-20849
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch,
> HIVE-20849.2.patch, HIVE-20849.3.patch, HIVE-20849.4.patch,
> HIVE-20849.5.patch, HIVE-20849.6.patch
>
>
> I was looking at this class because it blasts a lot of useless (to an admin)
> information to the logs. Especially if the table has a lot of columns, I see
> big blocks of logging that are meaningless to me. I request that the logging
> is toned down to debug, and some other improvements to the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.22.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-685) add UDFquote
[
https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760856#comment-16760856
]
Hive QA commented on HIVE-685:
--
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957614/HIVE.685.09.PATCH
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 888 failed/errored test(s), 7719 tests
executed
*Failed tests:*
{noformat}
TestAcidInputFormat - did not produce a TEST-*.xml file (likely timed out)
(batchId=300)
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=241)
TestAcidTableSerializer - did not produce a TEST-*.xml file (likely timed out)
(batchId=216)
TestAcidTableSetup - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestActivePassiveHA - did not produce a TEST-*.xml file (likely timed out)
(batchId=261)
TestAddResource - did not produce a TEST-*.xml file (likely timed out)
(batchId=324)
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=255)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=240)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out)
(batchId=254)
TestAvroGenericRecordReader - did not produce a TEST-*.xml file (likely timed
out) (batchId=300)
TestAvroHCatLoader - did not produce a TEST-*.xml file (likely timed out)
(batchId=207)
TestAvroHCatStorer - did not produce a TEST-*.xml file (likely timed out)
(batchId=207)
TestBasicStats - did not produce a TEST-*.xml file (likely timed out)
(batchId=296)
TestBeeLineExceptionHandling - did not produce a TEST-*.xml file (likely timed
out) (batchId=201)
TestBeeLineHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=201)
TestBeeLineOpts - did not produce a TEST-*.xml file (likely timed out)
(batchId=201)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=257)
TestBeelineArgParsing - did not produce a TEST-*.xml file (likely timed out)
(batchId=201)
TestBeelineConnectionUsingHiveSite - did not produce a TEST-*.xml file (likely
timed out) (batchId=257)
TestBeelinePasswordOption - did not produce a TEST-*.xml file (likely timed
out) (batchId=257)
TestBeelineSiteParser - did not produce a TEST-*.xml file (likely timed out)
(batchId=201)
TestBeelineWithUserHs2ConnectionFile - did not produce a TEST-*.xml file
(likely timed out) (batchId=256)
TestBlockedUdf - did not produce a TEST-*.xml file (likely timed out)
(batchId=286)
TestBucketIdResolverImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=216)
TestBufferedRows - did not produce a TEST-*.xml file (likely timed out)
(batchId=201)
TestCBOMaxNumToCNF - did not produce a TEST-*.xml file (likely timed out)
(batchId=305)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=265)
TestCLIServiceConnectionLimits - did not produce a TEST-*.xml file (likely
timed out) (batchId=233)
TestCLIServiceRestore - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestCachedStoreUpdateUsingEvents - did not produce a TEST-*.xml file (likely
timed out) (batchId=240)
TestCleaner - did not produce a TEST-*.xml file (likely timed out) (batchId=296)
TestCleaner2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=296)
TestCleanerWithReplication - did not produce a TEST-*.xml file (likely timed
out) (batchId=243)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=254)
TestCliDriverMethods - did not produce a TEST-*.xml file (likely timed out)
(batchId=200)
TestClientCommandHookFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=201)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=256)
TestColumnAccess - did not produce a TEST-*.xml file (likely timed out)
(batchId=305)
TestCommands - did not produce a TEST-*.xml file (likely timed out)
(batchId=201)
TestCommands - did not produce a TEST-*.xml file (likely timed out)
(batchId=204)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=242)
TestConditionalResolverCommonJoin - did not produce a TEST-*.xml file (likely
timed out) (batchId=325)
TestContext - did not produce a TEST-*.xml file (likely timed out) (batchId=311)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestCounterMapping - did not produce a TEST-*.xml file (likely timed out)
(batchId=324)
TestCreateMacroDesc - did not produce a TEST-*.xml file (likely timed out)
(batchId=325)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=243)
TestCrudCompactorOn
[jira] [Updated] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David McGinnis updated HIVE-21009:
--
Status: In Progress (was: Patch Available)
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0, 2.1.1, 2.1.0
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David McGinnis updated HIVE-21009:
--
Attachment: HIVE-21009.05.patch
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0, 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David McGinnis updated HIVE-21009:
--
Status: Patch Available (was: In Progress)
Fixing the style issues found in the last test. Not sure why the compiles
failed, error messages seem to indicate that the common changes weren't
compiled, but the output says it compiled the common changes first, and the
compile worked elsewhere. Will see if fixing the style issues fixes this before
raising the issue elsewhere.
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0, 2.1.1, 2.1.0
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21000: --- Status: Patch Available (was: In Progress) > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21215) Read Parquet INT64 timestamp
[ https://issues.apache.org/jira/browse/HIVE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-21215: > Read Parquet INT64 timestamp > > > Key: HIVE-21215 > URL: https://issues.apache.org/jira/browse/HIVE-21215 > Project: Hive > Issue Type: New Feature >Reporter: Karen Coppage >Assignee: Marta Kuczora >Priority: Major > > [WIP] > This patch enables Hive to start reading timestamps from Parquet written with > the new semantics: > With Parquet version 1.11, a new timestamp LogicalType with base INT64 and > the following metadata is introduced: > * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC > (aka Instant semantics) or not (LocalDateTime semantics). > * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp > Upon reading, the semantics of these new timestamps will be determined by > their metadata, while the semantics of INT96 timestamps will continue to be > deduced from the writer metadata. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21216) Write Parquet INT64 timestamp
[ https://issues.apache.org/jira/browse/HIVE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-21216: > Write Parquet INT64 timestamp > - > > Key: HIVE-21216 > URL: https://issues.apache.org/jira/browse/HIVE-21216 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: breaking_change > > [WIP] > This patch enables Hive to start writing int64 timestamps in Parquet. > With Parquet version 1.11, a new timestamp LogicalType with base INT64 and > the following metadata is introduced: > boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka > Instant semantics) or not (LocalDateTime semantics) > enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp > The timestamp will have LocalDateTime semantics (not converted to UTC). > Time unit (granularity) will be determined by the user. Default is > milliseconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21215) Read Parquet INT64 timestamp
[ https://issues.apache.org/jira/browse/HIVE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21215: - Description: [WIP] This patch enables Hive to start reading timestamps from Parquet written with the new semantics: With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced: * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics). * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata. This feature will be behind a flag for now. was: [WIP] This patch enables Hive to start reading timestamps from Parquet written with the new semantics: With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced: * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics). * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata. > Read Parquet INT64 timestamp > > > Key: HIVE-21215 > URL: https://issues.apache.org/jira/browse/HIVE-21215 > Project: Hive > Issue Type: New Feature >Reporter: Karen Coppage >Assignee: Marta Kuczora >Priority: Major > > [WIP] > This patch enables Hive to start reading timestamps from Parquet written with > the new semantics: > With Parquet version 1.11, a new timestamp LogicalType with base INT64 and > the following metadata is introduced: > * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC > (aka Instant semantics) or not (LocalDateTime semantics). > * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp > Upon reading, the semantics of these new timestamps will be determined by > their metadata, while the semantics of INT96 timestamps will continue to be > deduced from the writer metadata. > This feature will be behind a flag for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760885#comment-16760885
]
Hive QA commented on HIVE-21214:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15944/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15944/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15944/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[
https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760912#comment-16760912
]
Naveen Gangam commented on HIVE-20849:
--
[~belugabehr] Thanks for cleaning up some of these logging messages. I see that
the proposed fix unwraps the logging code from {{isXEnabled}} checks. Pardon my
ignorance with the recent changes. Are these checks no longer considered an
optimization to prevent the [potentially unnecessary] computation in generating
log strings (concatenation, etc)?
I realize in this case, its all string concatenation (without any expensive
method calls or exception stacks etc) but still there is some memory overhead
in concatenating strings. I dont have performance numbers either way but given
that these are mostly DEBUG logging that we unwrapped, that is normally turned
off on all production clusters, could the unnecessary overhead (memory and
performance) be more than we anticipate? This class is a well used codepath.
Even if some implementations of Loggers have a builtin checks in
{{LOG.debug()}} methods that make these additional checks unnecessary, there
are probably other implementations that dont.
Was just thinking out loud. Thanks
> Review of ConstantPropagateProcFactory
> --
>
> Key: HIVE-20849
> URL: https://issues.apache.org/jira/browse/HIVE-20849
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch,
> HIVE-20849.2.patch, HIVE-20849.3.patch, HIVE-20849.4.patch,
> HIVE-20849.5.patch, HIVE-20849.6.patch
>
>
> I was looking at this class because it blasts a lot of useless (to an admin)
> information to the logs. Especially if the table has a lot of columns, I see
> big blocks of logging that are meaningless to me. I request that the logging
> is toned down to debug, and some other improvements to the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barnabas Maidics updated HIVE-20758:
Status: Open (was: Patch Available)
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch, Screen Shot
> 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',|
> | 'transient_lastDdlTime'='1539710410')|
> ++
> {code}
--
This message was sent by Atla
[jira] [Updated] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barnabas Maidics updated HIVE-20758:
Attachment: HIVE-20758.2.patch
Status: Patch Available (was: Open)
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch, Screen Shot
> 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',|
> | 'transient_lastDdlTime'='1539710410')|
> ++
> {co
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[
https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760938#comment-16760938
]
BELUGA BEHR commented on HIVE-20849:
[~ngangam]
Hive uses SLF4J. One of its features is the concept of parameterized logging.
You pass a log message String with markers "{}" and the value that will
populate that marker.
{code:java}
// personObject.toString() is called within the debug method, only if DEBUG
level is enabled
LOG.debug("Hello, my name is {}", personObject);
{code}
Less code, cleaner code, faster code when debugging is not enabled.
[https://www.slf4j.org/faq.html#logging_performance]
For something like WARN/ERROR, there's really no need to wrap it or do anything
special. There is very rarely a case where WARN and ERROR logging is not
enabled, and we should not be optimizing for a scenario where it is disabled.
> Review of ConstantPropagateProcFactory
> --
>
> Key: HIVE-20849
> URL: https://issues.apache.org/jira/browse/HIVE-20849
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch,
> HIVE-20849.2.patch, HIVE-20849.3.patch, HIVE-20849.4.patch,
> HIVE-20849.5.patch, HIVE-20849.6.patch
>
>
> I was looking at this class because it blasts a lot of useless (to an admin)
> information to the logs. Especially if the table has a lot of columns, I see
> big blocks of logging that are meaningless to me. I request that the logging
> is toned down to debug, and some other improvements to the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita reassigned HIVE-21217: - > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760939#comment-16760939
]
Hive QA commented on HIVE-21214:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957608/HIVE-21214.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15729 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15944/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15944/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15944/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957608 - PreCommit-HIVE-Build
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-685) add UDFquote
[
https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760940#comment-16760940
]
Hive QA commented on HIVE-685:
--
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957614/HIVE.685.09.PATCH
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15945/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15945/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15945/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12957614/HIVE.685.09.PATCH was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957614 - PreCommit-HIVE-Build
> add UDFquote
>
>
> Key: HIVE-685
> URL: https://issues.apache.org/jira/browse/HIVE-685
> Project: Hive
> Issue Type: New Feature
>Reporter: Namit Jain
>Assignee: Mani M
>Priority: Major
> Labels: todoc4.0, udf
> Fix For: 4.0.0
>
> Attachments: HIVE.685.02.PATCH, HIVE.685.03.PATCH, HIVE.685.04.PATCH,
> HIVE.685.05.PATCH, HIVE.685.06.PATCH, HIVE.685.07.PATCH, HIVE.685.09.PATCH,
> HIVE.685.PATCH
>
>
> add UDFquote
> look at
> http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html
> for details
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21219) Cleanup pom.xml remote repository references
[
https://issues.apache.org/jira/browse/HIVE-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-21219:
---
> Cleanup pom.xml remote repository references
>
>
> Key: HIVE-21219
> URL: https://issues.apache.org/jira/browse/HIVE-21219
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> some of them seem to be not needed anymore and if I enable "cache clearing"
> for ptest the datanucleus repository sometimes returns errors like:
> {code}
> [ERROR] Failed to execute goal on project hive-shims-common: Could not
> resolve dependencies for project
> org.apache.hive.shims:hive-shims-common:jar:4.0.0-SNAPSHOT: The following
> artifacts could not be resolved:
> org.codehaus.jackson:jackson-core-asl:jar:1.9.13,
> org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13: Could not find artifact
> org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in datanucleus
> (http://www.datanucleus.org/downloads/maven2) -> [Help 1]
> {code}
> It happens for different artifacts ; but always with the "datanucleus" remote
> repoository.
> https://issues.apache.org/jira/browse/HIVE-21001?focusedCommentId=16760283&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16760283
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760973#comment-16760973 ] Zoltan Haindrich commented on HIVE-21001: - patch#22 seems to be good - the last failure was due to some transient issue inside the kafkamini driver...rerunning tests overall changes: * many q.out changes; most of them are improvement * druid intervals are in some places more precise than before * added "maven-central" as a repository ; because it tries to download all the artifacts from the repositories in the order they are listed; and that means to first check with the "datanucleus" repository - which sometimes returns an error code which then fails even the maven build; I've filed a separate jira(HIVE-21219) to clean up the repositories in the pom.xml; because I don't think any of those repositories are needed anymore. * disable project merge in "HiveRelBuilder" to avoid adverse effects * it seems the floor issue is after will be present even after this upgrade(HIVE-21208) - I plan investigate it further what options we have to fix that [~ashutoshc], [~jcamachorodriguez] Could you please take a look? > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
[ https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760976#comment-16760976 ] Milan Baran commented on HIVE-21218: I've made a quick fix for this problem - https://github.com/apache/hive/pull/526 > KafkaSerDe doesn't support topics created via Confluent Avro serializer > --- > > Key: HIVE-21218 > URL: https://issues.apache.org/jira/browse/HIVE-21218 > Project: Hive > Issue Type: Bug > Components: kafka integration, Serializers/Deserializers >Affects Versions: 3.1.1 >Reporter: Milan Baran >Priority: Major > > According to [Google > groups|https://groups.google.com/forum/#!topic/confluent-platform/JYhlXN0u9_A] > the Confluent avro serialzier uses propertiary format for kafka value - > <4 bytes of schema ID> conforms to schema>. > This format does not cause any problem for Confluent kafka deserializer which > respect the format however for hive kafka handler its bit a problem to > correctly deserialize kafka value, because Hive uses custom deserializer from > bytes to objects and ignores kafka consumer ser/deser classes provided via > table property. > It would be nice to support Confluent format with magic byte. > Also it would be great to support Schema registry as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21219) Cleanup pom.xml remote repository references
[
https://issues.apache.org/jira/browse/HIVE-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761002#comment-16761002
]
BELUGA BEHR commented on HIVE-21219:
I'm not sure if this is related, but I also see things like this:
{code:java}
The POM for org.glassfish:javax.el:jar:3.0.1-b08-SNAPSHOT is missing, no
dependency information available
Downloading:
file:${project.basedir}/src/main/site/resources/repo/org/glassfish/javax.el/3.0.1-b11-SNAPSHOT/maven-metadata.xml
{code}
As you can see, mvn is looking in {{$\{project.basedir}}} for files. I have
that directory in my project space.
> Cleanup pom.xml remote repository references
>
>
> Key: HIVE-21219
> URL: https://issues.apache.org/jira/browse/HIVE-21219
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> some of them seem to be not needed anymore and if I enable "cache clearing"
> for ptest the datanucleus repository sometimes returns errors like:
> {code}
> [ERROR] Failed to execute goal on project hive-shims-common: Could not
> resolve dependencies for project
> org.apache.hive.shims:hive-shims-common:jar:4.0.0-SNAPSHOT: The following
> artifacts could not be resolved:
> org.codehaus.jackson:jackson-core-asl:jar:1.9.13,
> org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13: Could not find artifact
> org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in datanucleus
> (http://www.datanucleus.org/downloads/maven2) -> [Help 1]
> {code}
> It happens for different artifacts ; but always with the "datanucleus" remote
> repoository.
> https://issues.apache.org/jira/browse/HIVE-21001?focusedCommentId=16760283&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16760283
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761010#comment-16761010
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
28s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 16 unchanged - 2
fixed = 16 total (was 18) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15946/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15946/yetus/patch-asflicense-problems.txt
|
| modules | C: ql accumulo-handler hbase-handler . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15946/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>
[jira] [Commented] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761020#comment-16761020
]
Hive QA commented on HIVE-21009:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957653/HIVE-21009.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15947/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15947/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15947/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-05 16:53:23.660
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15947/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-05 16:53:23.663
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 313e49f HIVE-20699: Query based compactor for full CRUD Acid
tables (Vaibhav Gumashta reviewed by Eugene Koifman)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 313e49f HIVE-20699: Query based compactor for full CRUD Acid
tables (Vaibhav Gumashta reviewed by Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-05 16:53:24.629
+ rm -rf ../yetus_PreCommit-HIVE-Build-15947
+ mkdir ../yetus_PreCommit-HIVE-Build-15947
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15947
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15947/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:2705
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
with conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:221: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:238: trailing whitespace.
error: patch failed:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:2705
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
with conflicts.
U common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
warning: 2 lines add whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-15947
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957653 - PreCommit-HIVE-Build
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0, 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.22.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761019#comment-16761019
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957652/HIVE-21001.22.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15729 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_nulls]
(batchId=142)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15946/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15946/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15946/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957652 - PreCommit-HIVE-Build
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21210:
---
Attachment: HIVE-21210.3.patch
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped.
> * Common module for partitioning work. That is, allow for a generic
> framework for dividing work into partitions (i.e. batches)
> * Modify {{CombineHiveInputFormat}} to take advantage of both modules,
> performing its same duties in a more Java OO way that is currently implemented
> * Add a partitioning (batching) implementation that enforces partitioning of
> a {{Collection}} based on the natural log of the {{Collection}} size so that
> it scales more slowly than a simple 1:100 ratio.
> * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to
> drop it into other places. One of the things I will introduce here is a
> "direct thread" {{ExecutorService}} so that even if there is a configuration
> for a thread pool to be disabled, it will still use an {{ExecutorService}} so
> that the project can avoid logic like "if this function is services by a
> thread pool, use a {{ExecutorService}} (and remember to close it later!)
> otherwise, create a single thread" so that things like [HIVE-16949] can be
> avoided in the future. Everything will just use an {{ExecutorService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21210:
---
Status: Open (was: Patch Available)
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped.
> * Common module for partitioning work. That is, allow for a generic
> framework for dividing work into partitions (i.e. batches)
> * Modify {{CombineHiveInputFormat}} to take advantage of both modules,
> performing its same duties in a more Java OO way that is currently implemented
> * Add a partitioning (batching) implementation that enforces partitioning of
> a {{Collection}} based on the natural log of the {{Collection}} size so that
> it scales more slowly than a simple 1:100 ratio.
> * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to
> drop it into other places. One of the things I will introduce here is a
> "direct thread" {{ExecutorService}} so that even if there is a configuration
> for a thread pool to be disabled, it will still use an {{ExecutorService}} so
> that the project can avoid logic like "if this function is services by a
> thread pool, use a {{ExecutorService}} (and remember to close it later!)
> otherwise, create a single thread" so that things like [HIVE-16949] can be
> avoided in the future. Everything will just use an {{ExecutorService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21210:
---
Status: Patch Available (was: Open)
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped.
> * Common module for partitioning work. That is, allow for a generic
> framework for dividing work into partitions (i.e. batches)
> * Modify {{CombineHiveInputFormat}} to take advantage of both modules,
> performing its same duties in a more Java OO way that is currently implemented
> * Add a partitioning (batching) implementation that enforces partitioning of
> a {{Collection}} based on the natural log of the {{Collection}} size so that
> it scales more slowly than a simple 1:100 ratio.
> * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to
> drop it into other places. One of the things I will introduce here is a
> "direct thread" {{ExecutorService}} so that even if there is a configuration
> for a thread pool to be disabled, it will still use an {{ExecutorService}} so
> that the project can avoid logic like "if this function is services by a
> thread pool, use a {{ExecutorService}} (and remember to close it later!)
> otherwise, create a single thread" so that things like [HIVE-16949] can be
> avoided in the future. Everything will just use an {{ExecutorService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Status: Open (was: Patch Available)
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.1, 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Attachment: HIVE-21071.7.patch
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Status: Patch Available (was: Open)
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.1, 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20977) Lazy evaluate the table object in PreReadTableEvent to improve get_partition performance
[
https://issues.apache.org/jira/browse/HIVE-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761064#comment-16761064
]
Karthik Manamcheri commented on HIVE-20977:
---
Can we merge this [~pvary]?
> Lazy evaluate the table object in PreReadTableEvent to improve get_partition
> performance
>
>
> Key: HIVE-20977
> URL: https://issues.apache.org/jira/browse/HIVE-20977
> Project: Hive
> Issue Type: Improvement
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20977.1.patch, HIVE-20977.2.patch,
> HIVE-20977.3.patch, HIVE-20977.4.patch
>
>
> The PreReadTableEvent is generated for non-table operations (such as
> get_partitions), but only if there is an event listener attached. However,
> this is also not necessary if the event listener is not interested in the
> read table event.
> For example, the TransactionalValidationListener's onEvent looks like this
> {code:java}
> @Override
> public void onEvent(PreEventContext context) throws MetaException,
> NoSuchObjectException,
> InvalidOperationException {
> switch (context.getEventType()) {
> case CREATE_TABLE:
> handle((PreCreateTableEvent) context);
> break;
> case ALTER_TABLE:
> handle((PreAlterTableEvent) context);
> break;
> default:
> //no validation required..
> }
> }{code}
>
> Note that for read table events it is a no-op. The problem is that the
> get_table is evaluated when creating the PreReadTableEvent finally to be just
> ignored!
> Look at the code below.. {{getMS().getTable(..)}} is evaluated irrespective
> of if the listener uses it or not.
> {code:java}
> private void fireReadTablePreEvent(String catName, String dbName, String
> tblName)
> throws MetaException, NoSuchObjectException {
> if(preListeners.size() > 0) {
> // do this only if there is a pre event listener registered (avoid
> unnecessary
> // metastore api call)
> Table t = getMS().getTable(catName, dbName, tblName);
> if (t == null) {
> throw new NoSuchObjectException(TableName.getQualified(catName, dbName,
> tblName)
> + " table not found");
> }
> firePreEvent(new PreReadTableEvent(t, this));
> }
> }
> {code}
> This can be improved by using a {{Supplier}} and lazily evaluating the table
> when needed (once when the first time it is called, memorized after that).
> *Motivation*
> Whenever a partition call occurs (get_partition, etc.), we fire the
> PreReadTableEvent. This affects performance since it fetches the table even
> if it is not being used. This change will improve performance on the
> get_partition calls.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761073#comment-16761073
]
Hive QA commented on HIVE-21000:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957615/HIVE-21000.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 15729 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key2]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=267)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNullDbName[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNullTblName[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNullDbName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNullTblName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNullDbName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNullTblName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionSpecsNullTblName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionValuesEmptySchema[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsAllNullDbName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsAllNullTblName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByFilterNullDbName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByFilterNullTblName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNullDbName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNullTblName[Remote]
(batchId=220)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15948/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15948/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15948/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957615 - PreCommit-HIVE-Build
> Upgrade thrift to at least 0.10.0
> -
>
> Key: HIVE-21000
> URL: https://issues.apache.org/jira/browse/HIVE-21000
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-21000.01.patch, sampler_before.png
>
>
> I was looking into some compile profiles for tables with lots of columns; and
> it turned out that [thrift 0.9.3 is allocating a
> List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348]
> during every hashcode calculation; but luckily THRIFT-2877 is improving on
> that - so I propose to upgrade to at least 0.10.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761102#comment-16761102
]
Hive QA commented on HIVE-21000:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
5s{color} | {color:red} The patch has 286 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
52s{color} | {color:red} patch/standalone-metastore/metastore-common cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} patch/serde cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 6m
41s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 21m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15948/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15948/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15948/yetus/patch-findbugs-standalone-metastore_metastore-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15948/yetus/patch-findbugs-serde.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15948/yetus/patch-findbugs-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15948/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore standalone-metastore/metastore-common
service-rpc serde ql . U: . |
| Console output |
http://104.198.109.2
[jira] [Commented] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761109#comment-16761109
]
Hive QA commented on HIVE-20758:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
7s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15949/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15949/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15949/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch, Screen Shot
> 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
[jira] [Updated] (HIVE-21211) Upgrade jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HIVE-21211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21211: --- Status: Open (was: Patch Available) > Upgrade jetty version to 9.4.x > -- > > Key: HIVE-21211 > URL: https://issues.apache.org/jira/browse/HIVE-21211 > Project: Hive > Issue Type: Task >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-21211.1.patch, HIVE-21211.2.patch, > HIVE-21211.3.patch, HIVE-21211.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21211) Upgrade jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HIVE-21211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21211: --- Attachment: HIVE-21211.4.patch Status: Patch Available (was: Open) > Upgrade jetty version to 9.4.x > -- > > Key: HIVE-21211 > URL: https://issues.apache.org/jira/browse/HIVE-21211 > Project: Hive > Issue Type: Task >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-21211.1.patch, HIVE-21211.2.patch, > HIVE-21211.3.patch, HIVE-21211.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761135#comment-16761135
]
Hive QA commented on HIVE-20758:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957657/HIVE-20758.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15729 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=45)
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=254)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[1]
(batchId=214)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15949/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15949/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15949/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957657 - PreCommit-HIVE-Build
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch, Screen Shot
> 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> +---
[jira] [Commented] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761153#comment-16761153
]
Jason Dere commented on HIVE-21214:
---
I'm not totally sure about the decision to change duplicate filename resolution
from file size to task attempt number. If you just fixed the file size logic to
take directories into account this would allow the existing logic to work in
the directory case. With task attempts we might have to worry about if this
breaks any existing cases. If we are convinced that we just need to worry about
Tez execution then I guess this could work, but this does not work on M/R with
speculative execution.
In terms of code comments, might be better with RB, but I'll add comments here:
* For the comments at the top of compareTempOrDuplicateFiles(), add a comment
this this breaks speculative execution.
* getDirSize() may not be the best name - this is really getting the file
size, and doing so recursively in the case that the file turns out to be a
directory. So maybe getFileSizeRecursivey() or something.
* Log at debug level in getDirSize()
I still need to make sense of the parsing changes
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761160#comment-16761160
]
Hive QA commented on HIVE-21210:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
35s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} common: The patch generated 5 new + 0 unchanged - 0
fixed = 5 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 4 new + 10 unchanged - 46
fixed = 14 total (was 56) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} common generated 18 new + 27 unchanged - 0 fixed = 45
total (was 27) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15950/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15950/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15950/yetus/diff-checkstyle-ql.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15950/yetus/diff-javadoc-javadoc-common.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15950/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15950/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places
[jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21210:
---
Status: Patch Available (was: Open)
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch, HIVE-21210.4.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped.
> * Common module for partitioning work. That is, allow for a generic
> framework for dividing work into partitions (i.e. batches)
> * Modify {{CombineHiveInputFormat}} to take advantage of both modules,
> performing its same duties in a more Java OO way that is currently implemented
> * Add a partitioning (batching) implementation that enforces partitioning of
> a {{Collection}} based on the natural log of the {{Collection}} size so that
> it scales more slowly than a simple 1:100 ratio.
> * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to
> drop it into other places. One of the things I will introduce here is a
> "direct thread" {{ExecutorService}} so that even if there is a configuration
> for a thread pool to be disabled, it will still use an {{ExecutorService}} so
> that the project can avoid logic like "if this function is services by a
> thread pool, use a {{ExecutorService}} (and remember to close it later!)
> otherwise, create a single thread" so that things like [HIVE-16949] can be
> avoided in the future. Everything will just use an {{ExecutorService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21210:
---
Status: Open (was: Patch Available)
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch, HIVE-21210.4.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped.
> * Common module for partitioning work. That is, allow for a generic
> framework for dividing work into partitions (i.e. batches)
> * Modify {{CombineHiveInputFormat}} to take advantage of both modules,
> performing its same duties in a more Java OO way that is currently implemented
> * Add a partitioning (batching) implementation that enforces partitioning of
> a {{Collection}} based on the natural log of the {{Collection}} size so that
> it scales more slowly than a simple 1:100 ratio.
> * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to
> drop it into other places. One of the things I will introduce here is a
> "direct thread" {{ExecutorService}} so that even if there is a configuration
> for a thread pool to be disabled, it will still use an {{ExecutorService}} so
> that the project can avoid logic like "if this function is services by a
> thread pool, use a {{ExecutorService}} (and remember to close it later!)
> otherwise, create a single thread" so that things like [HIVE-16949] can be
> avoided in the future. Everything will just use an {{ExecutorService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761196#comment-16761196
]
Hive QA commented on HIVE-21210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957668/HIVE-21210.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 15731 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input42] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_3] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_disablecbo_1]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_disablecbo_3]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_int_type_promotion]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat15]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_vc] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppr_pushdown3]
(batchId=30)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15950/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15950/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15950/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957668 - PreCommit-HIVE-Build
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch, HIVE-21210.4.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped
[jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
[
https://issues.apache.org/jira/browse/HIVE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21210:
---
Attachment: HIVE-21210.4.patch
> CombineHiveInputFormat Thread Pool Sizing
> -
>
> Key: HIVE-21210
> URL: https://issues.apache.org/jira/browse/HIVE-21210
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch,
> HIVE-21210.3.patch, HIVE-21210.4.patch
>
>
> Threadpools.
> Hive uses threadpools in several different places and each implementation is
> a little different and requires different configurations. I think that Hive
> needs to reign in and standardize the way that threadpools are used and
> threadpools should scale automatically without manual configuration. At any
> given time, there are many hundreds of threads running in the HS2 as the
> number of simultaneous connections increases and they surely cause contention
> with one-another.
> Here is an example:
> {code:java|title=CombineHiveInputFormat.java}
> // max number of threads we can use to check non-combinable paths
> private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
> private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per
> query and there is not much scaling here, it's simply 1 thread : 100 files
> ratio. This implies that to process 5000 files, there are 50 threads, after
> that, 50 threads are still used. Many Hive jobs these days involve more than
> 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware
> specs increase, and 50 threads seems like a lot when a service must support
> up to 80 connections:
> [https://www.cloudera.com/documentation/enterprise/5/latest/topics/admin_hive_tuning.html]
> Not to mention, I have never seen a scenario where HS2 is running on a host
> all by itself and has the entire system dedicated to it. Therefore it should
> be more friendly and spin up fewer threads.
> I am attaching a patch here that provides a few features:
> * Common module that produces {{ExecutorService}} which caps the number of
> threads it spins up at the number of processors a host has. Keep in mind that
> a class may submit as much work units ({{Callables}} as they would like, but
> the number of threads in the pool is capped.
> * Common module for partitioning work. That is, allow for a generic
> framework for dividing work into partitions (i.e. batches)
> * Modify {{CombineHiveInputFormat}} to take advantage of both modules,
> performing its same duties in a more Java OO way that is currently implemented
> * Add a partitioning (batching) implementation that enforces partitioning of
> a {{Collection}} based on the natural log of the {{Collection}} size so that
> it scales more slowly than a simple 1:100 ratio.
> * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to
> drop it into other places. One of the things I will introduce here is a
> "direct thread" {{ExecutorService}} so that even if there is a configuration
> for a thread pool to be disabled, it will still use an {{ExecutorService}} so
> that the project can avoid logic like "if this function is services by a
> thread pool, use a {{ExecutorService}} (and remember to close it later!)
> otherwise, create a single thread" so that things like [HIVE-16949] can be
> avoided in the future. Everything will just use an {{ExecutorService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761226#comment-16761226
]
Hive QA commented on HIVE-21071:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 10 new + 132 unchanged - 27
fixed = 142 total (was 159) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} ql generated 0 new + 2304 unchanged - 3 fixed = 2304
total (was 2307) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15951/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15951/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15951/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15951/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change me
[jira] [Commented] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761252#comment-16761252
]
Hive QA commented on HIVE-21071:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957666/HIVE-21071.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15730 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15951/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15951/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15951/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957666 - PreCommit-HIVE-Build
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-21214:
--
Attachment: HIVE-21214.2.patch
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch, HIVE-21214.2.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21221) Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port is configured
[ https://issues.apache.org/jira/browse/HIVE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21221: > Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port > is configured > --- > > Key: HIVE-21221 > URL: https://issues.apache.org/jira/browse/HIVE-21221 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > > When HiveServer2 comes up, it skips the start of the WebUI if > 1) hive.in.test is set to true > AND > 2) the WebUI port is not specified or default (hive.server2.webui.port) > > Right now, on LLAP daemon start, it is only checked if hive is in test > (condition 1) above. > > The LLAP Daemon start up code (to skip WebUI creation) should be consistent > with HS2, therefore if a port is specified (other than the default), the > WebUI should also be started in test mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Attachment: HIVE-21071.8.patch
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch, HIVE-21071.8.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Status: Open (was: Patch Available)
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.1, 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch, HIVE-21071.8.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Status: Patch Available (was: Open)
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch, HIVE-21071.8.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David McGinnis updated HIVE-21009:
--
Attachment: HIVE-21009.06.patch
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0, 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.06.patch, HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David McGinnis updated HIVE-21009:
--
Status: Patch Available (was: In Progress)
New patch after merging in the recent changes to HiveConf.java.
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0, 2.1.1, 2.1.0
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.06.patch, HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21009) LDAP - Specify binddn for ldap-search
[
https://issues.apache.org/jira/browse/HIVE-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David McGinnis updated HIVE-21009:
--
Status: In Progress (was: Patch Available)
> LDAP - Specify binddn for ldap-search
> -
>
> Key: HIVE-21009
> URL: https://issues.apache.org/jira/browse/HIVE-21009
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0, 2.1.1, 2.1.0
>Reporter: Thomas Uhren
>Assignee: David McGinnis
>Priority: Major
> Labels: features, newbie, security
> Attachments: HIVE-21009.01.patch, HIVE-21009.02.patch,
> HIVE-21009.03.patch, HIVE-21009.04.patch, HIVE-21009.05.patch,
> HIVE-21009.06.patch, HIVE-21009.patch
>
>
> When user accounts cannot do an LDAP search, there is currently no way of
> specifying a custom binddn to use for the ldap-search.
> So I'm missing something like that:
> {code}
> hive.server2.authentication.ldap.bindn=cn=ldapuser,ou=user,dc=example
> hive.server2.authentication.ldap.bindnpw=password
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Affects Version/s: (was: 3.1.1)
(was: 3.0.0)
3.2.0
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.2.patch,
> HIVE-21071.3.patch, HIVE-21071.4.patch, HIVE-21071.5.patch,
> HIVE-21071.6.patch, HIVE-21071.7.patch, HIVE-21071.8.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21205) Tests for replace flag in insert event messages in Metastore notifications.
[ https://issues.apache.org/jira/browse/HIVE-21205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-21205: Priority: Minor (was: Major) > Tests for replace flag in insert event messages in Metastore notifications. > --- > > Key: HIVE-21205 > URL: https://issues.apache.org/jira/browse/HIVE-21205 > Project: Hive > Issue Type: Test >Reporter: Bharathkrishna Guruvayoor Murali >Priority: Minor > > The replace flag is initially added in HIVE-16197. It would be good to have > some tests in TestDbNotificationListener to validate if the flag is set as > expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21221) Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port is configured
[ https://issues.apache.org/jira/browse/HIVE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21221: - Attachment: HIVE-21221.patch Status: Patch Available (was: Open) > Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port > is configured > --- > > Key: HIVE-21221 > URL: https://issues.apache.org/jira/browse/HIVE-21221 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Attachments: HIVE-21221.patch > > > When HiveServer2 comes up, it skips the start of the WebUI if > 1) hive.in.test is set to true > AND > 2) the WebUI port is not specified or default (hive.server2.webui.port) > > Right now, on LLAP daemon start, it is only checked if hive is in test > (condition 1) above. > > The LLAP Daemon start up code (to skip WebUI creation) should be consistent > with HS2, therefore if a port is specified (other than the default), the > WebUI should also be started in test mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761294#comment-16761294
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
32s{color} | {color:blue} ql in master has 2307 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} ql: The patch generated 0 new + 16 unchanged - 2
fixed = 16 total (was 18) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15952/dev-support/hive-personality.sh
|
| git revision | master / 313e49f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15952/yetus/patch-asflicense-problems.txt
|
| modules | C: ql accumulo-handler hbase-handler . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15952/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>
[jira] [Commented] (HIVE-21221) Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port is configured
[ https://issues.apache.org/jira/browse/HIVE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761295#comment-16761295 ] Jaume M commented on HIVE-21221: +1 > Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port > is configured > --- > > Key: HIVE-21221 > URL: https://issues.apache.org/jira/browse/HIVE-21221 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Attachments: HIVE-21221.patch > > > When HiveServer2 comes up, it skips the start of the WebUI if > 1) hive.in.test is set to true > AND > 2) the WebUI port is not specified or default (hive.server2.webui.port) > > Right now, on LLAP daemon start, it is only checked if hive is in test > (condition 1) above. > > The LLAP Daemon start up code (to skip WebUI creation) should be consistent > with HS2, therefore if a port is specified (other than the default), the > WebUI should also be started in test mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761300#comment-16761300
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957667/HIVE-21001.22.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15729 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15952/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15952/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15952/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12957667 - PreCommit-HIVE-Build
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch,
> HIVE-21001.22.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21222) ACID: When there are no delete deltas skip finding min max keys
[ https://issues.apache.org/jira/browse/HIVE-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21222: > ACID: When there are no delete deltas skip finding min max keys > --- > > Key: HIVE-21222 > URL: https://issues.apache.org/jira/browse/HIVE-21222 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > We create an orc reader in VectorizedOrcAcidRowBatchReader.findMinMaxKeys > (which will read 16K footer) even for cases where delete deltas does not > exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761318#comment-16761318
]
Jason Dere commented on HIVE-21214:
---
Left one nit on RB. Otherwise +1
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch, HIVE-21214.2.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21222) ACID: When there are no delete deltas skip finding min max keys
[ https://issues.apache.org/jira/browse/HIVE-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21222: - Attachment: HIVE-21222.1.patch > ACID: When there are no delete deltas skip finding min max keys > --- > > Key: HIVE-21222 > URL: https://issues.apache.org/jira/browse/HIVE-21222 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21222.1.patch > > > We create an orc reader in VectorizedOrcAcidRowBatchReader.findMinMaxKeys > (which will read 16K footer) even for cases where delete deltas does not > exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21222) ACID: When there are no delete deltas skip finding min max keys
[ https://issues.apache.org/jira/browse/HIVE-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21222: - Status: Patch Available (was: Open) > ACID: When there are no delete deltas skip finding min max keys > --- > > Key: HIVE-21222 > URL: https://issues.apache.org/jira/browse/HIVE-21222 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21222.1.patch > > > We create an orc reader in VectorizedOrcAcidRowBatchReader.findMinMaxKeys > (which will read 16K footer) even for cases where delete deltas does not > exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21222) ACID: When there are no delete deltas skip finding min max keys
[ https://issues.apache.org/jira/browse/HIVE-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761322#comment-16761322 ] Prasanth Jayachandran commented on HIVE-21222: -- [~ekoifman] could you please review this small patch? > ACID: When there are no delete deltas skip finding min max keys > --- > > Key: HIVE-21222 > URL: https://issues.apache.org/jira/browse/HIVE-21222 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21222.1.patch > > > We create an orc reader in VectorizedOrcAcidRowBatchReader.findMinMaxKeys > (which will read 16K footer) even for cases where delete deltas does not > exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21205) Tests for replace flag in insert event messages in Metastore notifications.
[ https://issues.apache.org/jira/browse/HIVE-21205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-21205: Attachment: HIVE-21205.1.patch Status: Patch Available (was: Open) > Tests for replace flag in insert event messages in Metastore notifications. > --- > > Key: HIVE-21205 > URL: https://issues.apache.org/jira/browse/HIVE-21205 > Project: Hive > Issue Type: Test >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-21205.1.patch > > > The replace flag is initially added in HIVE-16197. It would be good to have > some tests in TestDbNotificationListener to validate if the flag is set as > expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21205) Tests for replace flag in insert event messages in Metastore notifications.
[ https://issues.apache.org/jira/browse/HIVE-21205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-21205: --- Assignee: Bharathkrishna Guruvayoor Murali > Tests for replace flag in insert event messages in Metastore notifications. > --- > > Key: HIVE-21205 > URL: https://issues.apache.org/jira/browse/HIVE-21205 > Project: Hive > Issue Type: Test >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-21205.1.patch > > > The replace flag is initially added in HIVE-16197. It would be good to have > some tests in TestDbNotificationListener to validate if the flag is set as > expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
