[
https://issues.apache.org/jira/browse/SPARK-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Liu updated SPARK-3544:
-------------------------------
Component/s: (was: Spark Core)
SQL
Description:
Bug description:
The thriftServer cannot release locks correctly in Zookeeper.
Once a thriftServer is started, the 1st SQL submited by Beeline or JDBC
requiring locks can be successfully computed.
However, the 2rd SQL requiring locks will be blocked. And the thriftServer log
shows: INFO Driver: <PERFLOG method=acquireReadWriteLocks>.
2 Tests to illustrate the problem:
Test 1:
(0) Start thriftServer & use beeline to connect to it.
(1) Switch database(require locks);(OK)
(2) Drop table(require locks) “drop table src"; (Blocked)
Test 2:
(0) Start thriftServer & use beeline to connect to it.
(1) Drop table(require locks) "drop table src"; (OK)
(2) Drop another table(require locks) "drop table src2;"(Blocked)
Basic Information:
Spark 1.1.0
Hadoop 2.0.0-cdh4.6.0
Compile Command:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests
package
hive-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://ns</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>ns</value>
</property>
<property>
<name>dfs.ha.namenodes.ns</name>
<value>machine01,machine02</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns.machine01</name>
<value>machine01:54310</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns.machine02</name>
<value>machine02:54310</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.ns</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/metastore</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive_user</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive_123</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>false</value>
</property>
<property>
<name>hive.support.concurrency</name>
<description>Enable Hive's Table Lock Manager Service</description>
<value>true</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>machine01,machine02,machine03</value>
<description>Zookeeper quorum used by Hive's Table Lock
Manager</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>Hive warehouse directory</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>machine01:8032</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.type</name>
<value>BLOCK</value>
</property>
<property>
<name>hive.exec.show.job.failure.debug.info</name>
<value>true</value>
<description>
If a job fails, whether to provide a link in the CLI to the task with the
most failures, along with debugging hints if applicable.
</description>
</property>
<property>
<name>hive.hwi.listen.host</name>
<value>0.0.0.0</value>
<description>This is the host address the Hive Web Interface will listen
on</description>
</property>
<property>
<name>hive.hwi.listen.port</name>
<value>9999</value>
<description>This is the port the Hive Web Interface will listen
on</description>
</property>
<property>
<name>hive.hwi.war.file</name>
<value>/lib/hive-hwi-0.10.0-cdh4.2.0.war</value>
<description>This is the WAR file with the jsp content for Hive Web
Interface</description>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///usr/lib/hbase/hbase-0.94.15-cdh4.6.0-security.jar,file:///usr/lib/zookeeper/zookeeper.jar</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>machine01,machine02,machine03</value>
</property>
<property>
<name>hive.cli.print.header</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
<description>In unsecure mode, setting this property to true will cause the
metastore to execute DFS operations using the client's reported user and group
permissions. Note that this property must be set on both the client and server
sides. Further note that its best effort. If client sets its to true and server
sets it to false, client setting will be ignored.</description>
</property>
<property>
<name>hive.security.authorization.enabled</name>
<value>true</value>
<description>enable or disable the hive client authorization</description>
</property>
<property>
<name>hive.metastore.authorization.storage.checks</name>
<value>true</value>
</property>
<property>
<name>hive.security.authorization.createtable.owner.grants</name>
<value>ALL</value>
<description>the privileges automatically granted to the owner whenever a
table gets created.
An example like "select,drop" will grant select and drop privilege to the
owner of the table</description>
</property>
</configuration>
was:
Bug description:
The thriftServer cannot release locks correctly in Zookeeper.
Once a thriftServer is started, the 1st SQL submited by Beeline or JDBC
requiring locks can be successfully computed.
However, the 2rd SQL requiring locks will be blocked. And the thriftServer log
shows: INFO Driver: <PERFLOG method=acquireReadWriteLocks>.
2 Tests to illustrate the problem:
Test 1:
(0) Start thriftServer & use beeline to connect to it.
(1) Switch database(require locks);(OK)
(2) Drop table(require locks) “drop table src"; (Blocked)
Test 2:
(0) Start thriftServer & use beeline to connect to it.
(1) Drop table(require locks) "drop table src"; (OK)
(2) Drop another table(require locks) "drop table src2;"(Blocked)
Basic Information:
Spark 1.1.0
Hadoop 2.0.0-cdh4.6.0
Compile Command:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests
package
> SparkSQL thriftServer cannot release locks correctly in Zookeeper
> -----------------------------------------------------------------
>
> Key: SPARK-3544
> URL: https://issues.apache.org/jira/browse/SPARK-3544
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.0
> Reporter: Patrick Liu
> Priority: Critical
>
> Bug description:
> The thriftServer cannot release locks correctly in Zookeeper.
> Once a thriftServer is started, the 1st SQL submited by Beeline or JDBC
> requiring locks can be successfully computed.
> However, the 2rd SQL requiring locks will be blocked. And the thriftServer
> log shows: INFO Driver: <PERFLOG method=acquireReadWriteLocks>.
> 2 Tests to illustrate the problem:
> Test 1:
> (0) Start thriftServer & use beeline to connect to it.
> (1) Switch database(require locks);(OK)
> (2) Drop table(require locks) “drop table src"; (Blocked)
> Test 2:
> (0) Start thriftServer & use beeline to connect to it.
> (1) Drop table(require locks) "drop table src"; (OK)
> (2) Drop another table(require locks) "drop table src2;"(Blocked)
> Basic Information:
> Spark 1.1.0
> Hadoop 2.0.0-cdh4.6.0
> Compile Command:
> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M
> -XX:ReservedCodeCacheSize=512m"
> mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests
> package
> hive-site.xml:
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://ns</value>
> </property>
> <property>
> <name>dfs.nameservices</name>
> <value>ns</value>
> </property>
>
> <property>
> <name>dfs.ha.namenodes.ns</name>
> <value>machine01,machine02</value>
> </property>
>
> <property>
> <name>dfs.namenode.rpc-address.ns.machine01</name>
> <value>machine01:54310</value>
>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns.machine02</name>
> <value>machine02:54310</value>
>
> </property>
>
> <property>
> <name>dfs.client.failover.proxy.provider.ns</name>
>
> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost:3306/metastore</value>
> <description>JDBC connect string for a JDBC metastore</description>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> <description>Driver class name for a JDBC metastore</description>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive_user</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>hive_123</value>
> </property>
> <property>
> <name>datanucleus.autoCreateSchema</name>
> <value>false</value>
> </property>
> <property>
> <name>datanucleus.autoCreateTables</name>
> <value>true</value>
> </property>
> <property>
> <name>datanucleus.fixedDatastore</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.support.concurrency</name>
> <description>Enable Hive's Table Lock Manager Service</description>
> <value>true</value>
> </property>
> <property>
> <name>hive.zookeeper.quorum</name>
> <value>machine01,machine02,machine03</value>
> <description>Zookeeper quorum used by Hive's Table Lock
> Manager</description>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/user/hive/warehouse</value>
> <description>Hive warehouse directory</description>
> </property>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine01:8032</value>
> </property>
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.output.fileoutputformat.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.output.fileoutputformat.compress.type</name>
> <value>BLOCK</value>
> </property>
> <property>
> <name>hive.exec.show.job.failure.debug.info</name>
> <value>true</value>
> <description>
> If a job fails, whether to provide a link in the CLI to the task with the
> most failures, along with debugging hints if applicable.
> </description>
> </property>
> <property>
> <name>hive.hwi.listen.host</name>
> <value>0.0.0.0</value>
> <description>This is the host address the Hive Web Interface will listen
> on</description>
> </property>
> <property>
> <name>hive.hwi.listen.port</name>
> <value>9999</value>
> <description>This is the port the Hive Web Interface will listen
> on</description>
> </property>
> <property>
> <name>hive.hwi.war.file</name>
> <value>/lib/hive-hwi-0.10.0-cdh4.2.0.war</value>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>
> </property>
> <property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///usr/lib/hbase/hbase-0.94.15-cdh4.6.0-security.jar,file:///usr/lib/zookeeper/zookeeper.jar</value>
>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>machine01,machine02,machine03</value>
> </property>
> <property>
> <name>hive.cli.print.header</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.execute.setugi</name>
> <value>true</value>
> <description>In unsecure mode, setting this property to true will cause
> the metastore to execute DFS operations using the client's reported user and
> group permissions. Note that this property must be set on both the client and
> server sides. Further note that its best effort. If client sets its to true
> and server sets it to false, client setting will be ignored.</description>
> </property>
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>true</value>
> <description>enable or disable the hive client
> authorization</description>
> </property>
> <property>
> <name>hive.metastore.authorization.storage.checks</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.createtable.owner.grants</name>
> <value>ALL</value>
> <description>the privileges automatically granted to the owner whenever a
> table gets created.
> An example like "select,drop" will grant select and drop privilege to the
> owner of the table</description>
> </property>
> </configuration>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
