[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243756#comment-15243756 ] Vaibhav Gumashta commented on HIVE-6535: Looks like TestJdbcWithMiniHS2 was disabled when the QA run for this patch happened. I ran it on latest master and it runs fine. Rest of the failures are unrelated. I'll commit shortly. > JDBC: async wait should happen during fetch for results > --- > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, > HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[
https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242413#comment-15242413
]
Hive QA commented on HIVE-6535:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798154/HIVE-6535.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9952 tests executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-script_pipe.q-unionDistinct_1.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7593/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7593/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7593/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798154 - PreCommit-HIVE-TRUNK-Build
> JDBC: async wait should happen during fetch for results
> ---
>
> Key: HIVE-6535
> URL: https://issues.apache.org/jira/browse/HIVE-6535
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 0.14.0, 1.2.1, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch,
> HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch
>
>
> The hive jdbc client waits query completion during execute() call. It would
> be better to block in the jdbc for completion when the results are being
> fetched.
> This way the application using hive jdbc driver can do other tasks while
> asynchronous query execution is happening, until it needs to fetch the result
> set.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236314#comment-15236314 ] Thejas M Nair commented on HIVE-6535: - +1 pending tests > JDBC: async wait should happen during fetch for results > --- > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, > HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[
https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235082#comment-15235082
]
Hive QA commented on HIVE-6535:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12797833/HIVE-6535.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9970 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part_max
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7546/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7546/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7546/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12797833 - PreCommit-HIVE-TRUNK-Build
> JDBC: async wait should happen during fetch for results
> ---
>
> Key: HIVE-6535
> URL: https://issues.apache.org/jira/browse/HIVE-6535
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 0.14.0, 1.2.1, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch,
> HIVE-6535.4.patch
>
>
> The hive jdbc client waits query completion during execute() call. It would
> be better to block in the jdbc for completion when the results are being
> fetched.
> This way the application using hive jdbc driver can do other tasks while
> asynchronous query execution is happening, until it needs to fetch the result
> set.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[
https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233302#comment-15233302
]
Hive QA commented on HIVE-6535:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12797833/HIVE-6535.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9981 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7515/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7515/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7515/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12797833 - PreCommit-HIVE-TRUNK-Build
> JDBC: async wait should happen during fetch for results
> ---
>
> Key: HIVE-6535
> URL: https://issues.apache.org/jira/browse/HIVE-6535
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 0.14.0, 1.2.1, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch
>
>
> The hive jdbc client waits query completion during execute() call. It would
> be better to block in the jdbc for completion when the results are being
> fetched.
> This way the application using hive jdbc driver can do other tasks while
> asynchronous query execution is happening, until it needs to fetch the result
> set.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232548#comment-15232548 ] Thejas M Nair commented on HIVE-6535: - More details on the issue in waiting in fetch results and returning immediately after compile in stmt.execute, based on discussion with Vaibhav - * There is no fetch results when the query being executed is not a select query. We could block on calls like getUpdateCount(), but not everyone calls that. * In such cases, the user might call execute() followed by another Statement function such as execute, or close or cancel. It is not clear if the user would expect the previous query to block on this call after the first execute. Should the next call return an error if the first execute fails ? - That is not an intuitive behavior. * Statement.execute() documentation says ,that it throws SQLTimeoutException - when the driver has determined that the timeout value that was specified by the setQueryTimeout method has been exceeded and has at least attempted to cancel the currently running Statement. This cannot be implemented execute is not blocking. * Statement.cancel documentation talks about creating a separate thread for cancelling, another place where the assumption that stmt.execute is blocking is called out. > JDBC: async wait should happen during fetch for results > --- > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231519#comment-15231519 ] Vaibhav Gumashta commented on HIVE-6535: Discuss with [~thejas] and it makes sense to implement this as a non-standard api call. Looks like there is an expectation that Statement#execute be a blocking call. > JDBC: async wait should happen during fetch for results > --- > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[
https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223147#comment-15223147
]
Hive QA commented on HIVE-6535:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12796424/HIVE-6535.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 113 failed/errored test(s), 9936 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.hooks.TestHs2Hooks.testHookContexts
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
org.apache.hive.beeline.cli.TestHiveCli.testCmd
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions2
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource
org.apache.hive.jdbc.TestJdbcDriver2.setAutoCommitOnClosedConnection
org.apache.hive.jdbc.TestJdbcDriver2.testAutoCommit
org.apache.hive.jdbc.TestJdbcDriver2.testBadURL
org.apache.hive.jdbc.TestJdbcDriver2.testBuiltInUDFCol
org.apache.hive.jdbc.TestJdbcDriver2.testCloseResultSet
org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes
org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes2
org.apache.hive.jdbc.TestJdbcDriver2.testDatabaseMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testDescribeTable
org.apache.hive.jdbc.TestJdbcDriver2.testDriverProperties
org.apache.hive.jdbc.TestJdbcDriver2.testDuplicateColumnNameOrder
org.apache.hive.jdbc.TestJdbcDriver2.testErrorDiag
org.apache.hive.jdbc.TestJdbcDriver2.testErrorMessages
org.apache.hive.jdbc.TestJdbcDriver2.testExecutePreparedStatement
org.apache.hive.jdbc.TestJdbcDriver2.testExecuteQueryException
org.apache.hive.jdbc.TestJdbcDriver2.testExplainStmt
org.apache.hive.jdbc.TestJdbcDriver2.testExprCol
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstDfsCmds
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstError
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstNonMR
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstQuery
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstSetCmds
org.apache.hive.jdbc.TestJdbcDriver2.testGetQueryLog
org.apache.hive.jdbc.TestJdbcDriver2.testGetQueryLogOnDisabledLog
org.apache.hive.jdbc.TestJdbcDriver2.testImportedKeys
org.apache.hive.jdbc.TestJdbcDriver2.testIntervalTypes
org.apache.hive.jdbc.TestJdbcDriver2.testInvalidURL
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetCatalogs
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetClassicTableTypes
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetColumns
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetColumnsMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetExternalTables
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetHiveTableTypes
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetSchemas
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetTableTypes
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetTables
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetTablesClassic
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetTablesHive
org.apache.hive.jdbc.TestJdbcDriver2.testNonAsciiReturnValues
org.apache.hive.jdbc.TestJdbcDriver2.testNullResultSet
org.apache.hive.jdbc.TestJdbcDriver2.testNullType
org.apache.hive.jdbc.TestJdbcDriver2.testOutOfBoundCols
org.apache.hive.jdbc.TestJdbcDriver2.testParentReferences
org.apache.hive.jdbc.TestJdbcDriver2.testParseUrlHttpMode
org.apache.hive.jdbc.TestJdbcDriver2.testPostClose
org.apache.hive.jdbc.TestJdbcDriver2.testPrepareSetDate
org.apache.hive.jdbc.TestJdbcDriver2.testPrepareSetTimestamp
org.apache.hive.jdbc.TestJdbcDriver2.testPrepareStatement
org.apache.hive.jdbc.TestJdbcDriver2.testPrimaryKeys
org.apache.hive.jdbc.TestJdbcDriver2.testProcCols
org.apache.hive.jdbc.TestJdbcDriver2.testProccedures
org.apache.hive.jdbc.TestJdbcDriver2.testQueryCancel
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetColumnNameCaseInsensitive
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaDataDuplicateColumnNames
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAll
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllFetchSize
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllFromView
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllMaxRows
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllPartioned
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221207#comment-15221207 ] Thejas M Nair commented on HIVE-6535: - The setting of isLogBeingGenerated in HiveStatement has to be handled. Looks like it would need to be moved to HiveQueryResultSet. Note that HiveStatement.hasMoreLogs is documented as a public method, so that would still need to be around. > JDBC: async wait should happen during fetch for results > --- > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
