[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328307#comment-16328307
]
Hive QA commented on HIVE-18350:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12906327/HIVE-18350.5.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 75 failed/errored test(s), 11565 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_text] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_data_rename]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_fs] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_fs_overwrite]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_orc] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_orc_part]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[offset_limit_global_optimizer]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[load_fs2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[index_bitmap3]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[index_bitmap_auto]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[load_fs2]
(batchId=179)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[index_bitmap3]
(batchId=92)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[index_bitmap_auto]
(batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[bucket_mapjoin_mismatch1]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_data_into_acid]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_orc_negative2]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_orc_negative_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=219)
org.apache.hadoop.hive.ql.TestAcidOnTez.testInsertWithRemoveUnion (batchId=222)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversionVectorized
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket
(batchId=278)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket
(batchId=278)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testCopyExistingFilesOnDifferentFileSystem[0]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testCopyExistingFilesOnDifferentFileSystem[15]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testCopyNewFilesOnDifferentFileSystem[0]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testCopyNewFilesOnDifferentFileSystem[15]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testRenameExistingFilesOnSameFileSystem[0]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testRenameExistingFilesOnSameFileSystem[15]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testRenameNewFilesOnSameFileSystem[0]
(batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveCopyFiles.testRenameNewFilesOnSameFileSystem[15]
(batchId=278)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenateTable
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDumpLimit
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropPartitionedTable
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropUnpartitionedTable
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertToPartition
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInserts
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testInsertOverwriteOnPartitionedTableWithCM
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testInsertToMultiKeyPartition
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenamePartitionWithCM
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenameTableWithCM
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncateTable
(batchId=226)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncateWithCM
(batchId=226)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8649/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8649/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8649/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 75 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12906327 - PreCommit-HIVE-Build
> load data should rename files consistent with insert statements
> ---------------------------------------------------------------
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
> Reporter: Deepak Jaiswal
> Assignee: Deepak Jaiswal
> Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch,
> HIVE-18350.3.patch, HIVE-18350.4.patch, HIVE-18350.5.patch
>
>
> Insert statements create files of format ending with 0000_0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios and may
> cause trouble for other types of queries in future.
> We need consistent naming convention.
> For non-bucketed table, hive renames all the files regardless of how they
> were named by the user.
> For bucketed table, hive relies on user to name the files matching the
> bucket in non-strict mode. Hive assumes that the data belongs to same bucket
> in a file. In strict mode, loading bucketed table is disabled.
> This will likely affect most of the tests which load data which is pretty
> significant due to which it is further divided into two subtasks for smoother
> merge.
> For existing tables in customer database, it is recommended to reload
> bucketed tables otherwise if customer tries to run SMB join and there is a
> bucket for which there is no split, then there is a possibility of getting
> incorrect results. However, this is not a regression as it would happen even
> without the patch.
> With this patch however, and reloading data, the results should be correct.
> For non-bucketed tables and external tables, there is no difference in
> behavior and reloading data is not needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
