[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[ https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429289#comment-15429289 ] Matt McCline commented on HIVE-13874: - Committed to master and branch-2.1 > Tighten up EOF checking in Fast DeserializeRead classes; display better > exception information; add new Unit Tests > - > > Key: HIVE-13874 > URL: https://issues.apache.org/jira/browse/HIVE-13874 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch, > HIVE-13874.03.patch, HIVE-13874.04.patch, HIVE-13874.05.patch > > > Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond > stated row end are never read. Use WritableUtils.decodeVIntSize to check for > room ahead like regular LazyBinary code does. > Display more detailed information when an exception is thrown by > DeserializeRead classes. > Add Unit Tests, including some designed that catch the errors like HIVE-13818. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[ https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429287#comment-15429287 ] Matt McCline commented on HIVE-13874: - Test failures are unrelated. > Tighten up EOF checking in Fast DeserializeRead classes; display better > exception information; add new Unit Tests > - > > Key: HIVE-13874 > URL: https://issues.apache.org/jira/browse/HIVE-13874 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch, > HIVE-13874.03.patch, HIVE-13874.04.patch, HIVE-13874.05.patch > > > Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond > stated row end are never read. Use WritableUtils.decodeVIntSize to check for > room ahead like regular LazyBinary code does. > Display more detailed information when an exception is thrown by > DeserializeRead classes. > Add Unit Tests, including some designed that catch the errors like HIVE-13818. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[
https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429286#comment-15429286
]
Hive QA commented on HIVE-13874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12824660/HIVE-13874.05.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10470 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_2]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[load_dyn_part1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[transform_ppr1]
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/944/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/944/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-944/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12824660 - PreCommit-HIVE-MASTER-Build
> Tighten up EOF checking in Fast DeserializeRead classes; display better
> exception information; add new Unit Tests
> -
>
> Key: HIVE-13874
> URL: https://issues.apache.org/jira/browse/HIVE-13874
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch,
> HIVE-13874.03.patch, HIVE-13874.04.patch, HIVE-13874.05.patch
>
>
> Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond
> stated row end are never read. Use WritableUtils.decodeVIntSize to check for
> room ahead like regular LazyBinary code does.
> Display more detailed information when an exception is thrown by
> DeserializeRead classes.
> Add Unit Tests, including some designed that catch the errors like HIVE-13818.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[ https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428780#comment-15428780 ] Sergey Shelukhin commented on HIVE-13874: - +1 > Tighten up EOF checking in Fast DeserializeRead classes; display better > exception information; add new Unit Tests > - > > Key: HIVE-13874 > URL: https://issues.apache.org/jira/browse/HIVE-13874 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch, > HIVE-13874.03.patch, HIVE-13874.04.patch > > > Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond > stated row end are never read. Use WritableUtils.decodeVIntSize to check for > room ahead like regular LazyBinary code does. > Display more detailed information when an exception is thrown by > DeserializeRead classes. > Add Unit Tests, including some designed that catch the errors like HIVE-13818. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[ https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427420#comment-15427420 ] Sergey Shelukhin commented on HIVE-13874: - couple comments on RB > Tighten up EOF checking in Fast DeserializeRead classes; display better > exception information; add new Unit Tests > - > > Key: HIVE-13874 > URL: https://issues.apache.org/jira/browse/HIVE-13874 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch, > HIVE-13874.03.patch > > > Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond > stated row end are never read. Use WritableUtils.decodeVIntSize to check for > room ahead like regular LazyBinary code does. > Display more detailed information when an exception is thrown by > DeserializeRead classes. > Add Unit Tests, including some designed that catch the errors like HIVE-13818. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[
https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404753#comment-15404753
]
Hive QA commented on HIVE-13874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821477/HIVE-13874.03.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10452 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/738/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/738/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821477 - PreCommit-HIVE-MASTER-Build
> Tighten up EOF checking in Fast DeserializeRead classes; display better
> exception information; add new Unit Tests
> -
>
> Key: HIVE-13874
> URL: https://issues.apache.org/jira/browse/HIVE-13874
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch,
> HIVE-13874.03.patch
>
>
> Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond
> stated row end are never read. Use WritableUtils.decodeVIntSize to check for
> room ahead like regular LazyBinary code does.
> Display more detailed information when an exception is thrown by
> DeserializeRead classes.
> Add Unit Tests, including some designed that catch the errors like HIVE-13818.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[ https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404757#comment-15404757 ] Matt McCline commented on HIVE-13874: - Test failures are unrelated. > Tighten up EOF checking in Fast DeserializeRead classes; display better > exception information; add new Unit Tests > - > > Key: HIVE-13874 > URL: https://issues.apache.org/jira/browse/HIVE-13874 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch, > HIVE-13874.03.patch > > > Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond > stated row end are never read. Use WritableUtils.decodeVIntSize to check for > room ahead like regular LazyBinary code does. > Display more detailed information when an exception is thrown by > DeserializeRead classes. > Add Unit Tests, including some designed that catch the errors like HIVE-13818. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[
https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310721#comment-15310721
]
Hive QA commented on HIVE-13874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12807026/HIVE-13874.02.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10222 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLocksInSubquery
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/483/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/483/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-483/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12807026 - PreCommit-HIVE-MASTER-Build
> Tighten up EOF checking in Fast DeserializeRead classes; display better
> exception information; add new Unit Tests
> -
>
> Key: HIVE-13874
> URL: https://issues.apache.org/jira/browse/HIVE-13874
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13874.01.patch, HIVE-13874.02.patch
>
>
> Tighten up EOF bounds checking in LazyBinaryDeserializeRead so bytes beyond
> stated row end are never read.
> Display more detailed information when an exception is thrown by
> DeserializeRead classes.
> Add Unit Tests, including some that catch the error in HIVE-13818.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13874) Tighten up EOF checking in Fast DeserializeRead classes; display better exception information; add new Unit Tests
[
https://issues.apache.org/jira/browse/HIVE-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305692#comment-15305692
]
Hive QA commented on HIVE-13874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806579/HIVE-13874.01.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 125 failed/errored test(s), 9161 tests
executed
*Failed tests:*
{noformat}
TestAccumuloConnectionParameters - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-empty_join.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-cte_5.q-insert_into1.q-and-12-more - did
not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-union2.q-bucket4.q-union8.q-and-12-more - did not produce
a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-transform_ppr2.q-and-7-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-bucketizedhiveinputformat.q-orc_mr_pathalias.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-join1.q-infer_bucket_sort_bucketed_table.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
- did not produce a TEST-*.xml file
TestNegativeMinimrCliDriver-udf_local_resource.q-mapreduce_stack_trace_turnoff_hadoop20.q-mapreduce_stack_trace.q-and-6-more
- did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2_noskew_multi_distinct.q-vectorization_10.q-list_bucket_dml_2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-escape_distributeby1.q-auto_sortmerge_join_7.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-script_pipe.q-stats12.q-auto_join24.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriv
