[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958640#comment-16958640 ] Marta Kuczora commented on HIVE-22330: -- Pushed to master. Thanks [~klcopp] for the patch. > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch, > HIVE-22330.03.patch, HIVE-22330.03.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957465#comment-16957465 ] Hive QA commented on HIVE-22330: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12983739/HIVE-22330.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17545 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/19114/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19114/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19114/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12983739 - PreCommit-HIVE-Build > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch, > HIVE-22330.03.patch, HIVE-22330.03.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957434#comment-16957434 ] Hive QA commented on HIVE-22330: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 26s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-19114/dev-support/hive-personality.sh | | git revision | master / f1e59e9 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: storage-api U: storage-api | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-19114/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch, > HIVE-22330.03.patch, HIVE-22330.03.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956512#comment-16956512 ] Hive QA commented on HIVE-22330: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12983622/HIVE-22330.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17545 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/19094/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19094/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19094/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12983622 - PreCommit-HIVE-Build > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch, > HIVE-22330.03.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956474#comment-16956474 ] Hive QA commented on HIVE-22330: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 23s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-19094/dev-support/hive-personality.sh | | git revision | master / 72094da | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: storage-api U: storage-api | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-19094/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch, > HIVE-22330.03.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956118#comment-16956118 ] Peter Vary commented on HIVE-22330: --- +1 pending tests > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch, > HIVE-22330.03.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953007#comment-16953007 ] Hive QA commented on HIVE-22330: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12983132/HIVE-22330.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17541 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/19019/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19019/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19019/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12983132 - PreCommit-HIVE-Build > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952961#comment-16952961 ] Hive QA commented on HIVE-22330: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 23s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 4s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-19019/dev-support/hive-personality.sh | | git revision | master / 5ce0b40 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-19019/yetus/patch-asflicense-problems.txt | | modules | C: storage-api U: storage-api | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-19019/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch, HIVE-22330.02.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952562#comment-16952562 ] Karen Coppage commented on HIVE-22330: -- [~attilapiros] Since we set the smallBuffer size with an integer, integer overflow can occur even before we try to allocate memory – that's what these errors catch. > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950277#comment-16950277 ] Attila Zsolt Piros commented on HIVE-22330: --- I think the intention to use *smallbuffers* was to avoid fragmented allocation. As you see when the *nextElemLength* is over the *MAX_SIZE_FOR_SMALL_BUFFER* (1MB) then there is a new buffer allocated for the data and in that case the only possible exception is OOM. So I think when the *smallbuffer* is chosen the same should be kept: no other exception should be thrown (beside the implicit OOM). > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949871#comment-16949871 ] Hive QA commented on HIVE-22330: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12982799/HIVE-22330.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17520 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18958/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18958/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18958/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12982799 - PreCommit-HIVE-Build > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22330) Maximize smallBuffer usage in BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949854#comment-16949854 ] Hive QA commented on HIVE-22330: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 24s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18958/dev-support/hive-personality.sh | | git revision | master / f16509a | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-18958/yetus/patch-asflicense-problems.txt | | modules | C: storage-api U: storage-api | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18958/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Maximize smallBuffer usage in BytesColumnVector > --- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22330.01.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)