[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265988#comment-15265988
]
Ashutosh Chauhan commented on HIVE-13512:
-
+1
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch, HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264476#comment-15264476
]
Peter Slawski commented on HIVE-13512:
--
[~gopalv], could you please confirm my above statement regarding the test
failures. I would like to know the next steps I need to take for getting this
patch in. Thank you!
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch, HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259608#comment-15259608
]
Peter Slawski commented on HIVE-13512:
--
The test failure appear not to be related to this patch. Between the two tests
run, the patch has not changed. The common test fail between the two runs is
TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3. However, this is
using TestMiniSparkOnYarnCliDriver which should not be creating TezWork
objects, which this patch only touches.
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch, HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256215#comment-15256215
]
Hive QA commented on HIVE-13512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800330/HIVE-13512.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 9949 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hive.hcatalog.api.TestHCatClient.testCreateTableLike
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException
org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
org.apache.hive.hcatalog.api.TestHCatClient.testOtherFailure
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244273#comment-15244273
]
Hive QA commented on HIVE-13512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798638/HIVE-13512.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9983 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7617/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7617/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7617/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798638 - PreCommit-HIVE-TRUNK-Build
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240633#comment-15240633
]
Gopal V commented on HIVE-13512:
LGTM - +1 tests pending.
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
