[
https://issues.apache.org/jira/browse/IMPALA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Deschler updated IMPALA-10331:
-----------------------------------
Description:
After executing a series of CREATE FUNCTION DDLs, the corresponding DROP
FUNCTION DDLs will fail intermittently. The pattern is that functions beyond
some point in the list are missing from Thrift object Database.parameters. It
seems that the event processor is not keeping up with the CREATE FUNCTION DDLs
and the the DROP FUNCTION DDLs are looking at an old version. See repro below.
Query: drop FUNCTION IF EXISTS test_udf(DECIMAL(38,38))
Query: drop FUNCTION IF EXISTS test_udf(TINYINT)
ERROR: IllegalStateException: null
Preconditions.checkState(boolean) line: 159
Db.removeFunction(Function) line: 333
CatalogServiceCatalog(Catalog).removeFunction(Function) line: 335
CatalogServiceCatalog.removeFunction(Function) line: 2031
CatalogOpExecutor.dropFunction(TDropFunctionParams, TDdlExecResponse) line:
2157
CatalogOpExecutor.execDdlRequest(TDdlExecRequest) line: 442
JniCatalog.execDdl(byte[]) line: 173
Repro:
Start impalad with local catalog:
$IMPALA_HOME/bin/start-impala-cluster.py
-catalogd_args="catalog_topic_mode=minimal --hms_event_polling_interval_s=1"
--impalad_args="use_local_catalog=true"
--statestored_args="-statestore_update_frequency_ms=100"
Compile UDF shared library from attachment:
g++ shared -I$IMPALA_HOME/toolchain/boost$IMPALA_BOOST_VERSION/include -o
test_udf.so test_udf.cc
Upload UDF shared library to HDFS:
hdfs dfs -copyFromLocal test_udf.so /tmp/test_udf.so
Execute SQL attachment in a loop:
while [ 1 ]; do impala-shell -f test_udf.sql | tee -a test_udf.out; done
was:
After executing a series of CREATE FUNCTION DDLs, the corresponding DROP
FUNCTION DDLs will fail intermittently. The pattern is that functions beyond
some point in the list are missing from Thrift object Database.parameters. It
seems that the event processor is not keeping up with the CREATE FUNCTION DDLs
and the the DROP FUNCTION DDLs are looking at an old version. See repro below.
Query: drop FUNCTION IF EXISTS test_udf(DECIMAL(38,38))
Query: drop FUNCTION IF EXISTS test_udf(TINYINT)
ERROR: IllegalStateException: null
Preconditions.checkState(boolean) line: 159
Db.removeFunction(Function) line: 333
CatalogServiceCatalog(Catalog).removeFunction(Function) line: 335
CatalogServiceCatalog.removeFunction(Function) line: 2031
CatalogOpExecutor.dropFunction(TDropFunctionParams, TDdlExecResponse) line:
2157
CatalogOpExecutor.execDdlRequest(TDdlExecRequest) line: 442
JniCatalog.execDdl(byte[]) line: 173
Repro:
Start impalad with local catalog:
$IMPALA_HOME/bin/start-impala-cluster.py
-catalogd_args="catalog_topic_mode=minimal --hms_event_polling_interval_s=1"
--impalad_args="use_local_catalog=true"
--statestored_args="-statestore_update_frequency_ms=100"
Compile UDF shared library from attachment:
g++ shared -I$IMPALA_HOME/toolchain/boost$IMPALA_BOOST_VERSION/include -o
test_udf.so test_udf.cc
Upload UDF shared library to HDFS:
hdfs dfs -copyFromLocal test_udf.so /tmp/test_udf.so
Execute SQL attachment in a loop:
while [ 1 ]; do is -i localhost:21001 -f ~/test_udf.sql | tee -a test_sql.out;
done
> Intermittent DROP FUNCTION ERROR: IllegalStateException: null
> --------------------------------------------------------------
>
> Key: IMPALA-10331
> URL: https://issues.apache.org/jira/browse/IMPALA-10331
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.4.0
> Reporter: Kurt Deschler
> Assignee: Kurt Deschler
> Priority: Major
> Attachments: test_udf.cc, test_udf.sql
>
>
> After executing a series of CREATE FUNCTION DDLs, the corresponding DROP
> FUNCTION DDLs will fail intermittently. The pattern is that functions beyond
> some point in the list are missing from Thrift object Database.parameters. It
> seems that the event processor is not keeping up with the CREATE FUNCTION
> DDLs and the the DROP FUNCTION DDLs are looking at an old version. See repro
> below.
> Query: drop FUNCTION IF EXISTS test_udf(DECIMAL(38,38))
> Query: drop FUNCTION IF EXISTS test_udf(TINYINT)
> ERROR: IllegalStateException: null
> Preconditions.checkState(boolean) line: 159
> Db.removeFunction(Function) line: 333
> CatalogServiceCatalog(Catalog).removeFunction(Function) line: 335
> CatalogServiceCatalog.removeFunction(Function) line: 2031
> CatalogOpExecutor.dropFunction(TDropFunctionParams, TDdlExecResponse) line:
> 2157
> CatalogOpExecutor.execDdlRequest(TDdlExecRequest) line: 442
> JniCatalog.execDdl(byte[]) line: 173
> Repro:
> Start impalad with local catalog:
> $IMPALA_HOME/bin/start-impala-cluster.py
> -catalogd_args="catalog_topic_mode=minimal --hms_event_polling_interval_s=1"
> --impalad_args="use_local_catalog=true"
> --statestored_args="-statestore_update_frequency_ms=100"
> Compile UDF shared library from attachment:
> g++ shared -I$IMPALA_HOME/toolchain/boost$IMPALA_BOOST_VERSION/include -o
> test_udf.so test_udf.cc
> Upload UDF shared library to HDFS:
> hdfs dfs -copyFromLocal test_udf.so /tmp/test_udf.so
> Execute SQL attachment in a loop:
> while [ 1 ]; do impala-shell -f test_udf.sql | tee -a test_udf.out; done
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
