[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r435964404
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, "0")
hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)
+hiveServer2.getServices.asScala.foreach {
+ case t: ThriftCLIService if t.getPortNumber != 0 =>
+serverPort = t.getPortNumber
Review comment:
Ah I see
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r435946737
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, "0")
hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)
+hiveServer2.getServices.asScala.foreach {
+ case t: ThriftCLIService if t.getPortNumber != 0 =>
+serverPort = t.getPortNumber
Review comment:
I run the tests with http mode locally with #28738 and
```sql
build/sbt "hive-thriftserver/test-only *HiveThriftHttpServerSuite" -Phive
-Phive-thriftserver -Dsbt.override.build.repos=true -Phive-2.3
```
I check the logs in `target/unit-tests.log`
```java
20/06/05 06:35:55.237 pool-1-thread-1 INFO AbstractService:
Service:ThriftHttpCLIService is started.
20/06/05 06:35:55.237 pool-1-thread-1 INFO AbstractService:
Service:HiveServer2 is started.
20/06/05 06:35:55.326 Thread-17 INFO Server: jetty-9.4.18.v20190429; built:
2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm
1.8.0_251-b08
20/06/05 06:35:55.358 Thread-17 INFO session: DefaultSessionIdManager
workerName=node0
20/06/05 06:35:55.358 Thread-17 INFO session: No SessionScavenger set, using
defaults
20/06/05 06:35:55.359 Thread-17 INFO session: node0 Scavenging every 66ms
20/06/05 06:35:55.366 Thread-17 INFO ContextHandler: Started
o.e.j.s.ServletContextHandler@23f4b16a{/,null,AVAILABLE}
20/06/05 06:35:55.438 Thread-17 INFO AbstractConnector: Started
ServerConnector@1b76f67f{HTTP/1.1,[http/1.1]}{0.0.0.0:55923}
20/06/05 06:35:55.438 Thread-17 INFO Server: Started @7043ms
20/06/05 06:35:55.438 Thread-17 INFO ThriftCLIService: Started
ThriftHttpCLIService in http mode on port 55923 path=/cliservice/* with 5...500
worker threads
20/06/05 06:35:55.442 pool-1-thread-1 INFO Utils: Supplied authorities:
localhost:0
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: * JDBC param
deprecation *
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: The use of
hive.server2.transport.mode is deprecated.
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: Please use transportMode
like so: jdbc:hive2://:/dbName;transportMode=
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: * JDBC param
deprecation *
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: The use of
hive.server2.thrift.http.path is deprecated.
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: Please use httpPath like
so: jdbc:hive2://:/dbName;httpPath=
20/06/05 06:35:55.442 pool-1-thread-1 INFO Utils: Resolved authority:
localhost:0
20/06/05 06:35:55.663 pool-1-thread-1 ERROR HiveConnection: Error opening
session
org.apache.thrift.transport.TTransportException:
org.apache.http.conn.HttpHostConnectException: Connect to localhost:80
[localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
(Connection refused)
at
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:316)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:162)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:154)
at
org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:680)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:200)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at
org.apache.spark.sql.hive.thriftserver.SharedThriftServer.$anonfun$withMultipleConnectionJdbcStatement$1(SharedThriftServer.scala:106)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r435946737
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, "0")
hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)
+hiveServer2.getServices.asScala.foreach {
+ case t: ThriftCLIService if t.getPortNumber != 0 =>
+serverPort = t.getPortNumber
Review comment:
I run the tests with http mode locally with
```sql
build/sbt "hive-thriftserver/test-only *HiveThriftHttpServerSuite" -Phive
-Phive-thriftserver -Dsbt.override.build.repos=true -Phive-2.3
```
I check the logs in `target/unit-tests.log`
```java
20/06/05 06:35:55.237 pool-1-thread-1 INFO AbstractService:
Service:ThriftHttpCLIService is started.
20/06/05 06:35:55.237 pool-1-thread-1 INFO AbstractService:
Service:HiveServer2 is started.
20/06/05 06:35:55.326 Thread-17 INFO Server: jetty-9.4.18.v20190429; built:
2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm
1.8.0_251-b08
20/06/05 06:35:55.358 Thread-17 INFO session: DefaultSessionIdManager
workerName=node0
20/06/05 06:35:55.358 Thread-17 INFO session: No SessionScavenger set, using
defaults
20/06/05 06:35:55.359 Thread-17 INFO session: node0 Scavenging every 66ms
20/06/05 06:35:55.366 Thread-17 INFO ContextHandler: Started
o.e.j.s.ServletContextHandler@23f4b16a{/,null,AVAILABLE}
20/06/05 06:35:55.438 Thread-17 INFO AbstractConnector: Started
ServerConnector@1b76f67f{HTTP/1.1,[http/1.1]}{0.0.0.0:55923}
20/06/05 06:35:55.438 Thread-17 INFO Server: Started @7043ms
20/06/05 06:35:55.438 Thread-17 INFO ThriftCLIService: Started
ThriftHttpCLIService in http mode on port 55923 path=/cliservice/* with 5...500
worker threads
20/06/05 06:35:55.442 pool-1-thread-1 INFO Utils: Supplied authorities:
localhost:0
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: * JDBC param
deprecation *
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: The use of
hive.server2.transport.mode is deprecated.
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: Please use transportMode
like so: jdbc:hive2://:/dbName;transportMode=
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: * JDBC param
deprecation *
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: The use of
hive.server2.thrift.http.path is deprecated.
20/06/05 06:35:55.442 pool-1-thread-1 WARN Utils: Please use httpPath like
so: jdbc:hive2://:/dbName;httpPath=
20/06/05 06:35:55.442 pool-1-thread-1 INFO Utils: Resolved authority:
localhost:0
20/06/05 06:35:55.663 pool-1-thread-1 ERROR HiveConnection: Error opening
session
org.apache.thrift.transport.TTransportException:
org.apache.http.conn.HttpHostConnectException: Connect to localhost:80
[localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
(Connection refused)
at
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:316)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:162)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:154)
at
org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:680)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:200)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at
org.apache.spark.sql.hive.thriftserver.SharedThriftServer.$anonfun$withMultipleConnectionJdbcStatement$1(SharedThriftServer.scala:106)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r435922679
##
File path:
sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
##
@@ -144,6 +144,9 @@ public void run() {
// TODO: check defaults: maxTimeout, keepalive, maxBodySize,
bodyRecieveDuration, etc.
// Finally, start the server
httpServer.start();
+ // In case HIVE_SERVER2_THRIFT_HTTP_PORT or
hive.server2.thrift.http.port is configured with
+ // 0 which represents any free port, we should set it to the actual one
+ portNum = connector.getLocalPort();
Review comment:
I verify this locally, it seems to be ok.
but in https://github.com/apache/spark/pull/28738, I just run into the same
issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r435919651
##
File path:
sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
##
@@ -144,6 +144,9 @@ public void run() {
// TODO: check defaults: maxTimeout, keepalive, maxBodySize,
bodyRecieveDuration, etc.
// Finally, start the server
httpServer.start();
+ // In case HIVE_SERVER2_THRIFT_HTTP_PORT or
hive.server2.thrift.http.port is configured with
+ // 0 which represents any free port, we should set it to the actual one
+ portNum = connector.getLocalPort();
Review comment:
```
kentyao@hulk ~/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200601
sbin/start-thriftserver.sh --conf spark.hadoop.hive.server2.thrift.http.port=0
--conf spark.hadoop.hive.server2.transport.mode=http
starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging
to
/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200601/logs/spark-kentyao-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-hulk.local.out
kentyao@hulk ~/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200601 tail -f
-2
/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200601/logs/spark-kentyao-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-hulk.local.out
20/06/05 21:26:20 INFO HiveThriftServer2: HiveThriftServer2 started
20/06/05 21:26:20 INFO ThriftCLIService: Started ThriftHttpCLIService in
http mode on port 54379 path=/cliservice/* with 5...500 worker threads
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r432217803
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
Review comment:
thanks.
the `HTTP` mode is not used here, so the `.._THRIFT_PORT` is enough
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r431021323
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)
+hiveServer2.getServices.asScala.foreach {
+ case t: ThriftCLIService if t.getPortNumber != 0 =>
+serverPort = t.getPortNumber
+logInfo(s"Started HiveThriftServer2: port=$serverPort,
attempt=$attempt")
+ case _ =>
Review comment:
> shall we log something if we fail to bind the port?
there will be a requirement here
https://github.com/apache/spark/pull/28651/files#diff-ee6e4132bcb9b33369bd5a3567937607R58
after this.
Maybe enough for checking？
I think it might be unnecessary to log for each failed round.
> or fail here?
We can not fail in this match-case. this is also for other services
registered. or maybe we can add another case for matching - `case t:
ThriftCLIService =>`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r431021323
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)
+hiveServer2.getServices.asScala.foreach {
+ case t: ThriftCLIService if t.getPortNumber != 0 =>
+serverPort = t.getPortNumber
+logInfo(s"Started HiveThriftServer2: port=$serverPort,
attempt=$attempt")
+ case _ =>
Review comment:
> shall we log something if we fail to bind the port?
there will be a requirement here
https://github.com/apache/spark/pull/28651/files#diff-ee6e4132bcb9b33369bd5a3567937607R58
after this.
Maybe enough for checking？
I think it might be unnecessary to log for each failed round.
> or fail here?
We can not fail in this match-case. this is also for other services
registered. or maybe we can add another case for matching - `case t:
ThriftCLIService =>`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r431021323
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,19 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+// Set the HIVE_SERVER2_THRIFT_PORT to 0, so it could randomly pick any
free port to use.
+// It's much more robust than set a random port generated by ourselves
ahead
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)
+hiveServer2.getServices.asScala.foreach {
+ case t: ThriftCLIService if t.getPortNumber != 0 =>
+serverPort = t.getPortNumber
+logInfo(s"Started HiveThriftServer2: port=$serverPort,
attempt=$attempt")
+ case _ =>
Review comment:
there will be a requirement here
https://github.com/apache/spark/pull/28651/files#diff-ee6e4132bcb9b33369bd5a3567937607R58
after this. I think it‘s unnecessary to log for each otherfailed round. fail
in this match-case is not right，this is also for other services registered.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r430975380
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,17 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
Review comment:
okay
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r430975352
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,17 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
Review comment:
okay
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,17 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
Review comment:
okay
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28651: [SPARK-31833][SQL][test-hive1.2] Set HiveThriftServer2 with actual port while configured 0
yaooqinn commented on a change in pull request #28651:
URL: https://github.com/apache/spark/pull/28651#discussion_r430953790
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala
##
@@ -73,11 +68,17 @@ trait SharedThriftServer extends SharedSparkSession {
}
}
- private def startThriftServer(port: Int, attempt: Int): Unit = {
-logInfo(s"Trying to start HiveThriftServer2: port=$port, attempt=$attempt")
+ private def startThriftServer(attempt: Int): Unit = {
+logInfo(s"Trying to start HiveThriftServer2:, attempt=$attempt")
val sqlContext = spark.newSession().sqlContext
-sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname,
port.toString)
+sqlContext.setConf(ConfVars.HIVE_SERVER2_THRIFT_PORT.varname, 0.toString)
Review comment:
Yes, it's based on `java.net.ServerSocket` to get the listening port.
You can also see this behavior in the PR description
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
