[jira] [Commented] (IMPALA-7245) test_kudu.py::test_kudu_insert() fails with "Error adding columns to Kudu table tbl_with_defaults"
[
https://issues.apache.org/jira/browse/IMPALA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709335#comment-16709335
]
Tianyi Wang commented on IMPALA-7245:
-
This is a typical network failure, then fail on retrial case. We've seen a
couple of those and haven't figured out the cause of the network failure in any
of them.
> test_kudu.py::test_kudu_insert() fails with "Error adding columns to Kudu
> table tbl_with_defaults"
> --
>
> Key: IMPALA-7245
> URL: https://issues.apache.org/jira/browse/IMPALA-7245
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> query_test.test_kudu.TestKuduOperations.test_kudu_insert hit the following
> error:
> {noformat}
> query_test/test_kudu.py:93: in test_kudu_insert
> self.run_test_case('QueryTest/kudu_insert', vector,
> use_db=unique_database)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:343: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:339: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:483: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: ImpalaRuntimeException: Error adding columns to Kudu table
> tbl_with_defaults
> E CAUSED BY: NonRecoverableException: The column already exists: j{noformat}
> This is very similar to what we saw in IMPALA-6107, which we couldn't
> reproduce.
> It seems to be related to our interaction with the Kudu client. The statement
> starts executing at 23:05:11.17. From impalad.INFO:
> {noformat}
> I0629 23:05:11.170009 20602 impala-beeswax-server.cc:54] query(): query=alter
> table tbl_with_defaults add columns (j int null, k int not null default
> 1){noformat}
> This results in actions in catalogd, but catalogd encounters a timeout in the
> AsyncKuduClient while processing this and then immediately hits the error:
> {noformat}
> I0629 23:05:11.212054 28109 AsyncKuduClient.java:1756] Invalidating location
> master-127.0.0.1:7051(127.0.0.1:7051) for tablet Kudu Master: [peer
> master-127.0.0.1:7051(127.0.0.1:7051)] encountered a read timeout; closing
> the channel
> I0629 23:05:11.288261 11518 jni-util.cc:230]
> org.apache.impala.common.ImpalaRuntimeException: Error adding columns to Kudu
> table tbl_with_defaults
> at
> org.apache.impala.service.KuduCatalogOpExecutor.alterKuduTable(KuduCatalogOpExecutor.java:499)
> at
> org.apache.impala.service.KuduCatalogOpExecutor.addColumn(KuduCatalogOpExecutor.java:412)
> at
> org.apache.impala.service.CatalogOpExecutor.alterKuduTable(CatalogOpExecutor.java:600)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:420)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:270)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: org.apache.kudu.client.NonRecoverableException: The column already
> exists: j
> at
> org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
> at
> org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:351)
> at org.apache.kudu.client.KuduClient.alterTable(KuduClient.java:141)
> at
> org.apache.impala.service.KuduCatalogOpExecutor.alterKuduTable(KuduCatalogOpExecutor.java:494)
> ... 5 more
> {noformat}
> The Kudu master sees the alter table request, then the connection is torn
> down, then a duplicate alter table request:
> {noformat}
> I0629 23:05:11.197782 16157 catalog_manager.cc:2086] Servicing AlterTable
> request from {username='jenkins'} at 127.0.0.1:34332:
> table { table_name: "impala::test_kudu_insert_ca9324f5.tbl_with_defaults" }
> alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "j" type:
> INT32 is_key: false is_nullable: true cfile_block_size: 0 } } }
> alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "k" type:
> INT32 is_key: false is_nullable: false read_default_value: "\020\'\000\000"
> cfile_block_size: 0 } } }
> W0629 23:05:11.218549
[jira] [Resolved] (IMPALA-2648) catalogd crashes when serialized messages are over 2 GB
[
https://issues.apache.org/jira/browse/IMPALA-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-2648.
-
Resolution: Fixed
Fix Version/s: Impala 2.12.0
This has already been fixed by IMPALA-5990
> catalogd crashes when serialized messages are over 2 GB
> ---
>
> Key: IMPALA-2648
> URL: https://issues.apache.org/jira/browse/IMPALA-2648
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.3.0
>Reporter: Silvius Rus
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: compute-stats, crash, downgraded
> Fix For: Impala 2.12.0
>
>
> We've seen a catalogd crash triggered by loading the metadata for a table
> with about 20K partitions and 77 columns that has incremental stats. It
> looks like the serialized message is over 2GB which is the Java max array
> size. Ideally we should catch this exception and fail the query that needs
> this table's metadata with an appropriate message.
> {code}
> I1107 06:47:56.641507 30252 jni-util.cc:177] java.lang.OutOfMemoryError:
> Requested array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187)
> at
> com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831)
> at
> com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543)
> at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
> at
> com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123)
> at
> com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969)
> at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848)
> at
> com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
> at
> com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395)
> at com.cloudera.impala.thrift.TTable.write(TTable.java:1209)
> at
> com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
> at
> com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
> at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
> at
> com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
> at
> com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
> at
> com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
> at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
> at
> com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110)
> {code}
> You can identify this issue by looking at the metastore database. Here is
> how to see the size of the incremental stats for table id 12345. The table
> with this value of 624 MB of incremental stats led to the catalogd crash
> shown above.
> {code}
> postgres=# select pg_size_pretty(sum(length("PARTITION_PARAMS"."PARAM_KEY") +
> length("PARTITION_PARAMS"."PARAM_VALUE"))) from "PARTITIONS",
> "PARTITION_PARAMS" where "PARTITIONS"."TBL_ID"=12345 and
> "PARTITIONS"."PART_ID" = "PARTITION_PARAMS"."PART_ID" and
> "PARTITION_PARAMS"."PARAM_KEY" LIKE 'impala_intermediate%';
> pg_size_pretty
>
> 624 MB
> (1 row)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-2648) catalogd crashes when serialized messages are over 2 GB
[
https://issues.apache.org/jira/browse/IMPALA-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-2648.
-
Resolution: Fixed
Fix Version/s: Impala 2.12.0
This has already been fixed by IMPALA-5990
> catalogd crashes when serialized messages are over 2 GB
> ---
>
> Key: IMPALA-2648
> URL: https://issues.apache.org/jira/browse/IMPALA-2648
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.3.0
>Reporter: Silvius Rus
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: compute-stats, crash, downgraded
> Fix For: Impala 2.12.0
>
>
> We've seen a catalogd crash triggered by loading the metadata for a table
> with about 20K partitions and 77 columns that has incremental stats. It
> looks like the serialized message is over 2GB which is the Java max array
> size. Ideally we should catch this exception and fail the query that needs
> this table's metadata with an appropriate message.
> {code}
> I1107 06:47:56.641507 30252 jni-util.cc:177] java.lang.OutOfMemoryError:
> Requested array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187)
> at
> com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831)
> at
> com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543)
> at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
> at
> com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123)
> at
> com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969)
> at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848)
> at
> com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
> at
> com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395)
> at com.cloudera.impala.thrift.TTable.write(TTable.java:1209)
> at
> com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
> at
> com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
> at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
> at
> com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
> at
> com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
> at
> com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
> at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
> at
> com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110)
> {code}
> You can identify this issue by looking at the metastore database. Here is
> how to see the size of the incremental stats for table id 12345. The table
> with this value of 624 MB of incremental stats led to the catalogd crash
> shown above.
> {code}
> postgres=# select pg_size_pretty(sum(length("PARTITION_PARAMS"."PARAM_KEY") +
> length("PARTITION_PARAMS"."PARAM_VALUE"))) from "PARTITIONS",
> "PARTITION_PARAMS" where "PARTITIONS"."TBL_ID"=12345 and
> "PARTITIONS"."PART_ID" = "PARTITION_PARAMS"."PART_ID" and
> "PARTITION_PARAMS"."PARAM_KEY" LIKE 'impala_intermediate%';
> pg_size_pretty
>
> 624 MB
> (1 row)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7448) Periodically evict recently unused table from catalogd
[ https://issues.apache.org/jira/browse/IMPALA-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7448. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Periodically evict recently unused table from catalogd > -- > > Key: IMPALA-7448 > URL: https://issues.apache.org/jira/browse/IMPALA-7448 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > > To limit the memory consumption of catalog, we should experiment with a > mechanism automatically evicting recently unused tables from catalogd. > Initial design: > - impalad to report periodically/asynchronously the set of catalog objects > that were accessed > - catalogd to record some kind of last access time > - catalogd to have some facility to scan over all catalog objects, collect > some number of not-recently-used ones (eg to reach a target amount of evicted > memory), and issue invalidate commands to itself > - no need to have exact LRU behavior -- to simplify, we probably shouldn't > try to do a classical LRU linked list between all catalog objects. > - initial patch probably just triggered manually. Discussed either running > this on a schedule or running this based on JMX GC notifications if we see > that the catalogd finished an old-gen GC and the old gen is more than some > target percentage full. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7606) Time based auto invalidation not working
[ https://issues.apache.org/jira/browse/IMPALA-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7606. - Resolution: Fixed Fix Version/s: Impala 3.1.0 IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator CatalogdTableInvalidator detects if a table is in a normal state using Table.isLoaded() function. This is wrong because if there is an error during the loading of a table, isLoaded() returns true. This patch checks if the table is an IncompleteTable instead. Also fixed a bug in tryInstallGcListener(). A test is added to test the memory-based invalidation. Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Reviewed-on: http://gerrit.cloudera.org:8080/11512 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Time based auto invalidation not working > > > Key: IMPALA-7606 > URL: https://issues.apache.org/jira/browse/IMPALA-7606 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Rituparna Agrawal >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > Attachments: catalogd.INFO > > > Set the invalidate_tables_timeout_s to 60 on both catalogd and all impalads. > Have tables populated. Catalogd is keeps throwing this exception. Tables are > not getting invalidated. > No other logs seen on impalad from CatalogTableInvalidator > [10:50 AM] Parna Agrawal: > [10:50 AM] Parna Agrawal: W0921 10:50:19.465893 13428 > CatalogdTableInvalidator.java:284] Unexpected exception thrown while > attempting to automatically invalidate tables. Will retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) > W0921 10:50:24.467514 13428 CatalogdTableInvalidator.java:284] Unexpected > exception thrown while attempting to automatically invalidate tables. Will > retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3040) test_caching_ddl failed with unexpected get_num_cache_requests
[
https://issues.apache.org/jira/browse/IMPALA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-3040.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
I haven't seen this for months. Closing it.
> test_caching_ddl failed with unexpected get_num_cache_requests
> --
>
> Key: IMPALA-3040
> URL: https://issues.apache.org/jira/browse/IMPALA-3040
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.5.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> This test failed on the integration job.
> http://sandbox.jenkins.cloudera.com/view/Impala/view/Builds%20-%202.5.0%20release/job/impala-cdh5.7.0-integration/3/testReport/junit/query_test.test_hdfs_caching/TestHdfsCachingDdl/test_caching_ddl_exec_optiondisable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0table_format__text_none_/
> {code}
> query_test.test_hdfs_caching.TestHdfsCachingDdl.test_caching_ddl[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
> table_format: text/none] (from pytest)
> Failing for the past 1 build (Since Failed#3 )
> Took 47 sec.
> add description
> Error Message
> assert 9 == 10 + where 10 = get_num_cache_requests()
> Stacktrace
> query_test/test_hdfs_caching.py:132: in test_caching_ddl
> assert num_entries_pre == get_num_cache_requests()
> E assert 9 == 10
> E+ where 10 = get_num_cache_requests()
> Standard Error
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `cachedb` cascade;
> -- executing against localhost:21000
> create database cachedb;
> -- executing against localhost:21000
> use functional;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> use cachedb;
> -- executing against localhost:21000
> create table cached_tbl_nopart (i int) cached in 'testPool';
> -- executing against localhost:21000
> insert into cached_tbl_nopart select 1;
> -- executing against localhost:21000
> select * from cached_tbl_nopart;
> -- executing against localhost:21000
> create table like_cached_tbl_nopart like cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats like_cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> alter table cached_tbl_nopart set uncached;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool' with replication = 9;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool';
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=1) set cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> insert into cached_tbl_part partition(j) values(3, 3), (4, 4);
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=3) set cached in 'testPool' with
>
[jira] [Resolved] (IMPALA-3040) test_caching_ddl failed with unexpected get_num_cache_requests
[
https://issues.apache.org/jira/browse/IMPALA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-3040.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
I haven't seen this for months. Closing it.
> test_caching_ddl failed with unexpected get_num_cache_requests
> --
>
> Key: IMPALA-3040
> URL: https://issues.apache.org/jira/browse/IMPALA-3040
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.5.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> This test failed on the integration job.
> http://sandbox.jenkins.cloudera.com/view/Impala/view/Builds%20-%202.5.0%20release/job/impala-cdh5.7.0-integration/3/testReport/junit/query_test.test_hdfs_caching/TestHdfsCachingDdl/test_caching_ddl_exec_optiondisable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0table_format__text_none_/
> {code}
> query_test.test_hdfs_caching.TestHdfsCachingDdl.test_caching_ddl[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
> table_format: text/none] (from pytest)
> Failing for the past 1 build (Since Failed#3 )
> Took 47 sec.
> add description
> Error Message
> assert 9 == 10 + where 10 = get_num_cache_requests()
> Stacktrace
> query_test/test_hdfs_caching.py:132: in test_caching_ddl
> assert num_entries_pre == get_num_cache_requests()
> E assert 9 == 10
> E+ where 10 = get_num_cache_requests()
> Standard Error
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `cachedb` cascade;
> -- executing against localhost:21000
> create database cachedb;
> -- executing against localhost:21000
> use functional;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> use cachedb;
> -- executing against localhost:21000
> create table cached_tbl_nopart (i int) cached in 'testPool';
> -- executing against localhost:21000
> insert into cached_tbl_nopart select 1;
> -- executing against localhost:21000
> select * from cached_tbl_nopart;
> -- executing against localhost:21000
> create table like_cached_tbl_nopart like cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats like_cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> alter table cached_tbl_nopart set uncached;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool' with replication = 9;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool';
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=1) set cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> insert into cached_tbl_part partition(j) values(3, 3), (4, 4);
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=3) set cached in 'testPool' with
>
[jira] [Resolved] (IMPALA-7448) Periodically evict recently unused table from catalogd
[ https://issues.apache.org/jira/browse/IMPALA-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7448. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Periodically evict recently unused table from catalogd > -- > > Key: IMPALA-7448 > URL: https://issues.apache.org/jira/browse/IMPALA-7448 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > > To limit the memory consumption of catalog, we should experiment with a > mechanism automatically evicting recently unused tables from catalogd. > Initial design: > - impalad to report periodically/asynchronously the set of catalog objects > that were accessed > - catalogd to record some kind of last access time > - catalogd to have some facility to scan over all catalog objects, collect > some number of not-recently-used ones (eg to reach a target amount of evicted > memory), and issue invalidate commands to itself > - no need to have exact LRU behavior -- to simplify, we probably shouldn't > try to do a classical LRU linked list between all catalog objects. > - initial patch probably just triggered manually. Discussed either running > this on a schedule or running this based on JMX GC notifications if we see > that the catalogd finished an old-gen GC and the old gen is more than some > target percentage full. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7675) The result of UpdateTableUsage() RPC is not correctly handled.
[
https://issues.apache.org/jira/browse/IMPALA-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7675.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC
UpdateTableUsage() is logically a one-way RPC and the status object in
TUpdateTableUsageResponse is set only if there is an error at RPC
layer. This patch fixes the incorrect error handling that leads to
NullPointerException in ImpaladTableUsageTracer.
Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b
Reviewed-on: http://gerrit.cloudera.org:8080/11603
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> The result of UpdateTableUsage() RPC is not correctly handled.
> --
>
> Key: IMPALA-7675
> URL: https://issues.apache.org/jira/browse/IMPALA-7675
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tianyi Wang
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> ImpalaTableUsageTracker.report() doesn't handle the result of
> UpdateTableUsage() RPC correctly and triggers NullpointerException:
> {noformat}
> W1003 11:07:39.252918 6910 ImpaladTableUsageTracker.java:116] Unable to
> report table usage information to catalog server.
> Java exception follows:
> java.lang.NullPointerException
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker.report(ImpaladTableUsageTracker.java:110)
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker.access$000(ImpaladTableUsageTracker.java:44)
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker$1.run(ImpaladTableUsageTracker.java:56)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7606) Time based auto invalidation not working
[ https://issues.apache.org/jira/browse/IMPALA-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7606. - Resolution: Fixed Fix Version/s: Impala 3.1.0 IMPALA-7606: Fix IllegalStateException in CatalogTableInvalidator CatalogdTableInvalidator detects if a table is in a normal state using Table.isLoaded() function. This is wrong because if there is an error during the loading of a table, isLoaded() returns true. This patch checks if the table is an IncompleteTable instead. Also fixed a bug in tryInstallGcListener(). A test is added to test the memory-based invalidation. Change-Id: If938a40434b00af516445152f88832ef55d0d0ce Reviewed-on: http://gerrit.cloudera.org:8080/11512 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Time based auto invalidation not working > > > Key: IMPALA-7606 > URL: https://issues.apache.org/jira/browse/IMPALA-7606 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Rituparna Agrawal >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > Attachments: catalogd.INFO > > > Set the invalidate_tables_timeout_s to 60 on both catalogd and all impalads. > Have tables populated. Catalogd is keeps throwing this exception. Tables are > not getting invalidated. > No other logs seen on impalad from CatalogTableInvalidator > [10:50 AM] Parna Agrawal: > [10:50 AM] Parna Agrawal: W0921 10:50:19.465893 13428 > CatalogdTableInvalidator.java:284] Unexpected exception thrown while > attempting to automatically invalidate tables. Will retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) > W0921 10:50:24.467514 13428 CatalogdTableInvalidator.java:284] Unexpected > exception thrown while attempting to automatically invalidate tables. Will > retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7675) The result of UpdateTableUsage() RPC is not correctly handled.
[
https://issues.apache.org/jira/browse/IMPALA-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7675.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
IMPALA-7675: Fix the error handling of UpdateTableUsage() RPC
UpdateTableUsage() is logically a one-way RPC and the status object in
TUpdateTableUsageResponse is set only if there is an error at RPC
layer. This patch fixes the incorrect error handling that leads to
NullPointerException in ImpaladTableUsageTracer.
Change-Id: Iccba4c6f4696ef08bc8a614ae13f62b5e445917b
Reviewed-on: http://gerrit.cloudera.org:8080/11603
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> The result of UpdateTableUsage() RPC is not correctly handled.
> --
>
> Key: IMPALA-7675
> URL: https://issues.apache.org/jira/browse/IMPALA-7675
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tianyi Wang
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> ImpalaTableUsageTracker.report() doesn't handle the result of
> UpdateTableUsage() RPC correctly and triggers NullpointerException:
> {noformat}
> W1003 11:07:39.252918 6910 ImpaladTableUsageTracker.java:116] Unable to
> report table usage information to catalog server.
> Java exception follows:
> java.lang.NullPointerException
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker.report(ImpaladTableUsageTracker.java:110)
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker.access$000(ImpaladTableUsageTracker.java:44)
> at
> org.apache.impala.catalog.ImpaladTableUsageTracker$1.run(ImpaladTableUsageTracker.java:56)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7532) Add retry/back-off to fetch-from-catalog RPCs
[ https://issues.apache.org/jira/browse/IMPALA-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7532. - Resolution: Fixed Fix Version/s: Impala 3.2.0 > Add retry/back-off to fetch-from-catalog RPCs > - > > Key: IMPALA-7532 > URL: https://issues.apache.org/jira/browse/IMPALA-7532 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.2.0 > > > Currently if there is an error connecting to the catalog server, the 'fetch > from catalog' implementation will retry with no apparent backoff. We should > retry for some period of time with backoff in between the attempts, so that > impala can ride over short interruptions of the catalog service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7532) Add retry/back-off to fetch-from-catalog RPCs
[ https://issues.apache.org/jira/browse/IMPALA-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7532. - Resolution: Fixed Fix Version/s: Impala 3.2.0 > Add retry/back-off to fetch-from-catalog RPCs > - > > Key: IMPALA-7532 > URL: https://issues.apache.org/jira/browse/IMPALA-7532 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.2.0 > > > Currently if there is an error connecting to the catalog server, the 'fetch > from catalog' implementation will retry with no apparent backoff. We should > retry for some period of time with backoff in between the attempts, so that > impala can ride over short interruptions of the catalog service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7767) Add metrics and profiles for automatic invalidation
Tianyi Wang created IMPALA-7767: --- Summary: Add metrics and profiles for automatic invalidation Key: IMPALA-7767 URL: https://issues.apache.org/jira/browse/IMPALA-7767 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.2.0 Reporter: Tianyi Wang Assignee: Tianyi Wang The automatic invalidation introduced by IMPALA-7448 lacks supportive metrics and query profiles. They should be added to help users to understand whether it's working and to diagnose possible issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7767) Add metrics and profiles for automatic invalidation
Tianyi Wang created IMPALA-7767: --- Summary: Add metrics and profiles for automatic invalidation Key: IMPALA-7767 URL: https://issues.apache.org/jira/browse/IMPALA-7767 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.2.0 Reporter: Tianyi Wang Assignee: Tianyi Wang The automatic invalidation introduced by IMPALA-7448 lacks supportive metrics and query profiles. They should be added to help users to understand whether it's working and to diagnose possible issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7383) Make METASTORE_DB configurable and default to escaped $IMPALA_HOME
[ https://issues.apache.org/jira/browse/IMPALA-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659764#comment-16659764 ] Tianyi Wang commented on IMPALA-7383: - [~arodoni_cloudera] For dev only > Make METASTORE_DB configurable and default to escaped $IMPALA_HOME > -- > > Key: IMPALA-7383 > URL: https://issues.apache.org/jira/browse/IMPALA-7383 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Trivial > Labels: infrastructure > Fix For: Impala 3.1.0 > > > Some developers keep multiple impala repos on their disk. Isolating > METASTORE_DB may help with switching between those repos without needing to > reload the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7670) Drop table with a concurrent refresh throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/IMPALA-7670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644081#comment-16644081
]
Tianyi Wang commented on IMPALA-7670:
-
Do you know more details about how it happened? I ran refresh and drop stats in
a loop and haven't reproduced.
> Drop table with a concurrent refresh throws ConcurrentModificationException
> ---
>
> Key: IMPALA-7670
> URL: https://issues.apache.org/jira/browse/IMPALA-7670
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: Tianyi Wang
>Priority: Major
>
> * This is a Catalog V2 only bug*.
> Saw this in the Catalog server.
> {noformat}
> I1004 16:38:55.236702 85380 jni-util.cc:308]
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1471)
> at
> org.apache.impala.catalog.FeFsTable$Utils.getPartitionFromThriftPartitionSpec(FeFsTable.java:407)
> at
> org.apache.impala.catalog.HdfsTable.getPartitionFromThriftPartitionSpec(HdfsTable.java:694)
> at
> org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:407)
> at
> org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:386)
> at
> org.apache.impala.service.CatalogOpExecutor.bulkAlterPartitions(CatalogOpExecutor.java:3193)
> at
> org.apache.impala.service.CatalogOpExecutor.dropTableStats(CatalogOpExecutor.java:1255)
> at
> org.apache.impala.service.CatalogOpExecutor.dropStats(CatalogOpExecutor.java:1148)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:301)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:157)
> {noformat}
> Still need to dig into it, but seems like something is off with locking
> somewhere.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7670) Drop table with a concurrent refresh throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/IMPALA-7670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644056#comment-16644056
]
Tianyi Wang commented on IMPALA-7670:
-
I think dropStats() is only called by drop stats statement, not by drop table
statement. Is it concurrent "drop stats" and "refresh"?
> Drop table with a concurrent refresh throws ConcurrentModificationException
> ---
>
> Key: IMPALA-7670
> URL: https://issues.apache.org/jira/browse/IMPALA-7670
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: Tianyi Wang
>Priority: Major
>
> * This is a Catalog V2 only bug*.
> Saw this in the Catalog server.
> {noformat}
> I1004 16:38:55.236702 85380 jni-util.cc:308]
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1471)
> at
> org.apache.impala.catalog.FeFsTable$Utils.getPartitionFromThriftPartitionSpec(FeFsTable.java:407)
> at
> org.apache.impala.catalog.HdfsTable.getPartitionFromThriftPartitionSpec(HdfsTable.java:694)
> at
> org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:407)
> at
> org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:386)
> at
> org.apache.impala.service.CatalogOpExecutor.bulkAlterPartitions(CatalogOpExecutor.java:3193)
> at
> org.apache.impala.service.CatalogOpExecutor.dropTableStats(CatalogOpExecutor.java:1255)
> at
> org.apache.impala.service.CatalogOpExecutor.dropStats(CatalogOpExecutor.java:1148)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:301)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:157)
> {noformat}
> Still need to dig into it, but seems like something is off with locking
> somewhere.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7675) The result of UpdateTableUsage() RPC is not correctly handled.
Tianyi Wang created IMPALA-7675:
---
Summary: The result of UpdateTableUsage() RPC is not correctly
handled.
Key: IMPALA-7675
URL: https://issues.apache.org/jira/browse/IMPALA-7675
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tianyi Wang
ImpalaTableUsageTracker.report() doesn't handle the result of
UpdateTableUsage() RPC correctly and triggers NullpointerException:
{noformat}
W1003 11:07:39.252918 6910 ImpaladTableUsageTracker.java:116] Unable to report
table usage information to catalog server.
Java exception follows:
java.lang.NullPointerException
at
org.apache.impala.catalog.ImpaladTableUsageTracker.report(ImpaladTableUsageTracker.java:110)
at
org.apache.impala.catalog.ImpaladTableUsageTracker.access$000(ImpaladTableUsageTracker.java:44)
at
org.apache.impala.catalog.ImpaladTableUsageTracker$1.run(ImpaladTableUsageTracker.java:56)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7675) The result of UpdateTableUsage() RPC is not correctly handled.
Tianyi Wang created IMPALA-7675:
---
Summary: The result of UpdateTableUsage() RPC is not correctly
handled.
Key: IMPALA-7675
URL: https://issues.apache.org/jira/browse/IMPALA-7675
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tianyi Wang
ImpalaTableUsageTracker.report() doesn't handle the result of
UpdateTableUsage() RPC correctly and triggers NullpointerException:
{noformat}
W1003 11:07:39.252918 6910 ImpaladTableUsageTracker.java:116] Unable to report
table usage information to catalog server.
Java exception follows:
java.lang.NullPointerException
at
org.apache.impala.catalog.ImpaladTableUsageTracker.report(ImpaladTableUsageTracker.java:110)
at
org.apache.impala.catalog.ImpaladTableUsageTracker.access$000(ImpaladTableUsageTracker.java:44)
at
org.apache.impala.catalog.ImpaladTableUsageTracker$1.run(ImpaladTableUsageTracker.java:56)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7663) count_user_privilege isn't 0 at the end of test_owner_privileges_without_grant
Tianyi Wang created IMPALA-7663:
---
Summary: count_user_privilege isn't 0 at the end of
test_owner_privileges_without_grant
Key: IMPALA-7663
URL: https://issues.apache.org/jira/browse/IMPALA-7663
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Adam Holley
{noformat}
08:35:35 === FAILURES
===
08:35:35 TestOwnerPrivileges.test_owner_privileges_without_grant[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
08:35:35 authorization/test_owner_privileges.py:340: in
test_owner_privileges_without_grant
08:35:35 sentry_refresh_timeout_s=SENTRY_REFRESH_TIMEOUT_S)
08:35:35 authorization/test_owner_privileges.py:378: in
__execute_owner_privilege_tests_oo_nogrant
08:35:35 assert self.count_user_privileges(result) == 0
08:35:35 E assert 1 == 0
08:35:35 E+ where 1 = ()
08:35:35 E+where =
.count_user_privileges
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7663) count_user_privilege isn't 0 at the end of test_owner_privileges_without_grant
Tianyi Wang created IMPALA-7663:
---
Summary: count_user_privilege isn't 0 at the end of
test_owner_privileges_without_grant
Key: IMPALA-7663
URL: https://issues.apache.org/jira/browse/IMPALA-7663
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Adam Holley
{noformat}
08:35:35 === FAILURES
===
08:35:35 TestOwnerPrivileges.test_owner_privileges_without_grant[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
08:35:35 authorization/test_owner_privileges.py:340: in
test_owner_privileges_without_grant
08:35:35 sentry_refresh_timeout_s=SENTRY_REFRESH_TIMEOUT_S)
08:35:35 authorization/test_owner_privileges.py:378: in
__execute_owner_privilege_tests_oo_nogrant
08:35:35 assert self.count_user_privileges(result) == 0
08:35:35 E assert 1 == 0
08:35:35 E+ where 1 = ()
08:35:35 E+where =
.count_user_privileges
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7662) test_parquet reads bad_magic_number.parquet without an error
Tianyi Wang created IMPALA-7662:
---
Summary: test_parquet reads bad_magic_number.parquet without an
error
Key: IMPALA-7662
URL: https://issues.apache.org/jira/browse/IMPALA-7662
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tim Armstrong
{noformat}
09:51:41 === FAILURES
===
09:51:41 TestParquet.test_parquet[exec_option: {'batch_size': 0, 'num_nodes':
0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action':
'HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:FAIL@0.5',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
09:51:41 [gw5] linux2 -- Python 2.7.5
/data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/bin/../infra/python/env/bin/python
09:51:41 query_test/test_scanners.py:300: in test_parquet
09:51:41 self.run_test_case('QueryTest/parquet', vector)
09:51:41 common/impala_test_suite.py:423: in run_test_case
09:51:41 assert False, "Expected exception: %s" % expected_str
09:51:41 E AssertionError: Expected exception: File
'hdfs://localhost:20500/test-warehouse/bad_magic_number_parquet/bad_magic_number.parquet'
has an invalid version number:
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7662) test_parquet reads bad_magic_number.parquet without an error
Tianyi Wang created IMPALA-7662:
---
Summary: test_parquet reads bad_magic_number.parquet without an
error
Key: IMPALA-7662
URL: https://issues.apache.org/jira/browse/IMPALA-7662
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tim Armstrong
{noformat}
09:51:41 === FAILURES
===
09:51:41 TestParquet.test_parquet[exec_option: {'batch_size': 0, 'num_nodes':
0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action':
'HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:FAIL@0.5',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
09:51:41 [gw5] linux2 -- Python 2.7.5
/data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/bin/../infra/python/env/bin/python
09:51:41 query_test/test_scanners.py:300: in test_parquet
09:51:41 self.run_test_case('QueryTest/parquet', vector)
09:51:41 common/impala_test_suite.py:423: in run_test_case
09:51:41 assert False, "Expected exception: %s" % expected_str
09:51:41 E AssertionError: Expected exception: File
'hdfs://localhost:20500/test-warehouse/bad_magic_number_parquet/bad_magic_number.parquet'
has an invalid version number:
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7661) test_reconnect is flaky in asan
Tianyi Wang created IMPALA-7661:
---
Summary: test_reconnect is flaky in asan
Key: IMPALA-7661
URL: https://issues.apache.org/jira/browse/IMPALA-7661
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
test_reconnect becomes flaky recently. The reason is that it waits 2 seconds
for the shell to connect to impalad. Extending the waiting time in asan should
work.
{noformat}
22:54:40 === FAILURES
===
22:54:40 __ TestImpalaShellInteractive.test_reconnect
___
22:54:40
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/shell/test_shell_interactive.py:236:
in test_reconnect
22:54:40 assert get_num_open_sessions(initial_impala_service) ==
num_sessions_initial + 1, \
22:54:40 E AssertionError: Not connected to localhost.localdomain:21000
22:54:40 E assert 1 == (1 + 1)
22:54:40 E+ where 1 = ()
22:54:40 === 2451 tests deselected by "-m 'execute_serially'"
===
22:54:40 1 failed, 190 passed, 10 skipped, 2451 deselected, 17 xfailed, 1
pytest-warnings in 2849.85 seconds
22:54:40 -- 2018-10-03 22:54:40,709 INFO MainThread: Found 3 impalad/1
statestored/1 catalogd process(es)
22:54:40
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7661) test_reconnect is flaky in asan
Tianyi Wang created IMPALA-7661:
---
Summary: test_reconnect is flaky in asan
Key: IMPALA-7661
URL: https://issues.apache.org/jira/browse/IMPALA-7661
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
test_reconnect becomes flaky recently. The reason is that it waits 2 seconds
for the shell to connect to impalad. Extending the waiting time in asan should
work.
{noformat}
22:54:40 === FAILURES
===
22:54:40 __ TestImpalaShellInteractive.test_reconnect
___
22:54:40
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/shell/test_shell_interactive.py:236:
in test_reconnect
22:54:40 assert get_num_open_sessions(initial_impala_service) ==
num_sessions_initial + 1, \
22:54:40 E AssertionError: Not connected to localhost.localdomain:21000
22:54:40 E assert 1 == (1 + 1)
22:54:40 E+ where 1 = ()
22:54:40 === 2451 tests deselected by "-m 'execute_serially'"
===
22:54:40 1 failed, 190 passed, 10 skipped, 2451 deselected, 17 xfailed, 1
pytest-warnings in 2849.85 seconds
22:54:40 -- 2018-10-03 22:54:40,709 INFO MainThread: Found 3 impalad/1
statestored/1 catalogd process(es)
22:54:40
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7532) Add retry/back-off to fetch-from-catalog RPCs
[ https://issues.apache.org/jira/browse/IMPALA-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631153#comment-16631153 ] Tianyi Wang commented on IMPALA-7532: - [~parna] Do you have the fatal message? > Add retry/back-off to fetch-from-catalog RPCs > - > > Key: IMPALA-7532 > URL: https://issues.apache.org/jira/browse/IMPALA-7532 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Tianyi Wang >Priority: Major > > Currently if there is an error connecting to the catalog server, the 'fetch > from catalog' implementation will retry with no apparent backoff. We should > retry for some period of time with backoff in between the attempts, so that > impala can ride over short interruptions of the catalog service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7639) impala-shell stuck at "Starting Impala Shell without Kerberos authentication" in test_multiline_queries_in_history
[
https://issues.apache.org/jira/browse/IMPALA-7639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang reassigned IMPALA-7639:
---
Assignee: Joe McDonnell
> impala-shell stuck at "Starting Impala Shell without Kerberos authentication"
> in test_multiline_queries_in_history
> --
>
> Key: IMPALA-7639
> URL: https://issues.apache.org/jira/browse/IMPALA-7639
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: broken-build, flaky
>
> Impala-shell wrote nothing other than "Starting Impala Shell without Kerberos
> authentication" to stdout until timeout.
> {noformat}
> 03:11:31 [gw3] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 03:11:31 shell/test_shell_interactive.py:303: in
> test_multiline_queries_in_history
> 03:11:31 child_proc.expect(PROMPT_REGEX)
> 03:11:31
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
> in expect
> 03:11:31 timeout, searchwindowsize)
> 03:11:31
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
> in expect_list
> 03:11:31 timeout, searchwindowsize)
> 03:11:31
> ../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
> in expect_loop
> 03:11:31 raise TIMEOUT(str(err) + '\n' + str(self))
> 03:11:31 E TIMEOUT: Timeout exceeded.
> 03:11:31 E
> 03:11:31 E version: 3.3
> 03:11:31 E command: /home/ubuntu/Impala/bin/impala-shell.sh
> 03:11:31 E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
> 03:11:31 E searcher:
> 03:11:31 E buffer (last 100 chars): 'Starting Impala Shell without Kerberos
> authentication\r\n'
> 03:11:31 E before (last 100 chars): 'Starting Impala Shell without Kerberos
> authentication\r\n'
> 03:11:31 E after:
> 03:11:31 E match: None
> 03:11:31 E match_index: None
> 03:11:31 E exitstatus: None
> 03:11:31 E flag_eof: False
> 03:11:31 E pid: 118020
> 03:11:31 E child_fd: 18
> 03:11:31 E closed: False
> 03:11:31 E timeout: 30
> 03:11:31 E delimiter:
> 03:11:31 E logfile: None
> 03:11:31 E logfile_read: None
> 03:11:31 E logfile_send: None
> 03:11:31 E maxread: 2000
> 03:11:31 E ignorecase: False
> 03:11:31 E searchwindowsize: None
> 03:11:31 E delaybeforesend: 0.05
> 03:11:31 E delayafterclose: 0.1
> 03:11:31 E delayafterterminate: 0.1
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3258/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7639) impala-shell stuck at "Starting Impala Shell without Kerberos authentication" in test_multiline_queries_in_history
Tianyi Wang created IMPALA-7639:
---
Summary: impala-shell stuck at "Starting Impala Shell without
Kerberos authentication" in test_multiline_queries_in_history
Key: IMPALA-7639
URL: https://issues.apache.org/jira/browse/IMPALA-7639
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Impala-shell wrote nothing other than "Starting Impala Shell without Kerberos
authentication" to stdout until timeout.
{noformat}
03:11:31 [gw3] linux2 -- Python 2.7.12
/home/ubuntu/Impala/bin/../infra/python/env/bin/python
03:11:31 shell/test_shell_interactive.py:303: in
test_multiline_queries_in_history
03:11:31 child_proc.expect(PROMPT_REGEX)
03:11:31
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
in expect
03:11:31 timeout, searchwindowsize)
03:11:31
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
in expect_list
03:11:31 timeout, searchwindowsize)
03:11:31
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
in expect_loop
03:11:31 raise TIMEOUT(str(err) + '\n' + str(self))
03:11:31 E TIMEOUT: Timeout exceeded.
03:11:31 E
03:11:31 E version: 3.3
03:11:31 E command: /home/ubuntu/Impala/bin/impala-shell.sh
03:11:31 E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
03:11:31 E searcher:
03:11:31 E buffer (last 100 chars): 'Starting Impala Shell without Kerberos
authentication\r\n'
03:11:31 E before (last 100 chars): 'Starting Impala Shell without Kerberos
authentication\r\n'
03:11:31 E after:
03:11:31 E match: None
03:11:31 E match_index: None
03:11:31 E exitstatus: None
03:11:31 E flag_eof: False
03:11:31 E pid: 118020
03:11:31 E child_fd: 18
03:11:31 E closed: False
03:11:31 E timeout: 30
03:11:31 E delimiter:
03:11:31 E logfile: None
03:11:31 E logfile_read: None
03:11:31 E logfile_send: None
03:11:31 E maxread: 2000
03:11:31 E ignorecase: False
03:11:31 E searchwindowsize: None
03:11:31 E delaybeforesend: 0.05
03:11:31 E delayafterclose: 0.1
03:11:31 E delayafterterminate: 0.1
{noformat}
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3258/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7639) impala-shell stuck at "Starting Impala Shell without Kerberos authentication" in test_multiline_queries_in_history
Tianyi Wang created IMPALA-7639:
---
Summary: impala-shell stuck at "Starting Impala Shell without
Kerberos authentication" in test_multiline_queries_in_history
Key: IMPALA-7639
URL: https://issues.apache.org/jira/browse/IMPALA-7639
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Impala-shell wrote nothing other than "Starting Impala Shell without Kerberos
authentication" to stdout until timeout.
{noformat}
03:11:31 [gw3] linux2 -- Python 2.7.12
/home/ubuntu/Impala/bin/../infra/python/env/bin/python
03:11:31 shell/test_shell_interactive.py:303: in
test_multiline_queries_in_history
03:11:31 child_proc.expect(PROMPT_REGEX)
03:11:31
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1451:
in expect
03:11:31 timeout, searchwindowsize)
03:11:31
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1466:
in expect_list
03:11:31 timeout, searchwindowsize)
03:11:31
../infra/python/env/local/lib/python2.7/site-packages/pexpect/__init__.py:1568:
in expect_loop
03:11:31 raise TIMEOUT(str(err) + '\n' + str(self))
03:11:31 E TIMEOUT: Timeout exceeded.
03:11:31 E
03:11:31 E version: 3.3
03:11:31 E command: /home/ubuntu/Impala/bin/impala-shell.sh
03:11:31 E args: ['/home/ubuntu/Impala/bin/impala-shell.sh']
03:11:31 E searcher:
03:11:31 E buffer (last 100 chars): 'Starting Impala Shell without Kerberos
authentication\r\n'
03:11:31 E before (last 100 chars): 'Starting Impala Shell without Kerberos
authentication\r\n'
03:11:31 E after:
03:11:31 E match: None
03:11:31 E match_index: None
03:11:31 E exitstatus: None
03:11:31 E flag_eof: False
03:11:31 E pid: 118020
03:11:31 E child_fd: 18
03:11:31 E closed: False
03:11:31 E timeout: 30
03:11:31 E delimiter:
03:11:31 E logfile: None
03:11:31 E logfile_read: None
03:11:31 E logfile_send: None
03:11:31 E maxread: 2000
03:11:31 E ignorecase: False
03:11:31 E searchwindowsize: None
03:11:31 E delaybeforesend: 0.05
03:11:31 E delayafterclose: 0.1
03:11:31 E delayafterterminate: 0.1
{noformat}
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3258/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7593) test_automatic_invalidation failing in S3
[
https://issues.apache.org/jira/browse/IMPALA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-7593:
Fix Version/s: Impala 3.1.0
> test_automatic_invalidation failing in S3
> -
>
> Key: IMPALA-7593
> URL: https://issues.apache.org/jira/browse/IMPALA-7593
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> Note that the build has the fix for IMPALA-7580
> {noformat}
> 4:59:01 ___ TestAutomaticCatalogInvalidation.test_v1_catalog
> ___
> 04:59:01 custom_cluster/test_automatic_invalidation.py:63: in test_v1_catalog
> 04:59:01 self._run_test(cursor)
> 04:59:01 custom_cluster/test_automatic_invalidation.py:58: in _run_test
> 04:59:01 assert time.time() < timeout
> 04:59:01 E assert 1537355634.805718 < 1537355634.394429
> 04:59:01 E+ where 1537355634.805718 = ()
> 04:59:01 E+where = time.time
> 04:59:01 Captured stderr setup
> -
> 04:59:01 -- 2018-09-19 04:13:22,796 INFO MainThread: Starting cluster
> with command:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py
> --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args="--invalidate_tables_timeout_s=20" '
> '--state_store_args="--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50" '
> '--catalogd_args="--invalidate_tables_timeout_s=20" '
> 04:59:01 04:13:23 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 04:59:01 04:13:23 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 04:59:01 04:13:24 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 04:59:01 04:13:25 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 04:59:01 04:13:26 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 04:59:01 04:13:29 MainThread: Found 3 impalad/1 statestored/1 catalogd
> process(es)
> 04:59:01 04:13:29 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:29 MainThread: Waiting for num_known_live_backends=3. Current
> value: 0
> 04:59:01 04:13:30 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:30 MainThread: Waiting for num_known_live_backends=3. Current
> value: 1
> 04:59:01 04:13:31 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:31 MainThread: Waiting for num_known_live_backends=3. Current
> value: 2
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25001
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25002
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Impala Cluster Running with 3 nodes (3
> coordinators, 3 executors).
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25010
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Getting
> num_known_live_backends from
>
[jira] [Resolved] (IMPALA-7593) test_automatic_invalidation failing in S3
[
https://issues.apache.org/jira/browse/IMPALA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7593.
-
Resolution: Fixed
> test_automatic_invalidation failing in S3
> -
>
> Key: IMPALA-7593
> URL: https://issues.apache.org/jira/browse/IMPALA-7593
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build
>
> Note that the build has the fix for IMPALA-7580
> {noformat}
> 4:59:01 ___ TestAutomaticCatalogInvalidation.test_v1_catalog
> ___
> 04:59:01 custom_cluster/test_automatic_invalidation.py:63: in test_v1_catalog
> 04:59:01 self._run_test(cursor)
> 04:59:01 custom_cluster/test_automatic_invalidation.py:58: in _run_test
> 04:59:01 assert time.time() < timeout
> 04:59:01 E assert 1537355634.805718 < 1537355634.394429
> 04:59:01 E+ where 1537355634.805718 = ()
> 04:59:01 E+where = time.time
> 04:59:01 Captured stderr setup
> -
> 04:59:01 -- 2018-09-19 04:13:22,796 INFO MainThread: Starting cluster
> with command:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py
> --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args="--invalidate_tables_timeout_s=20" '
> '--state_store_args="--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50" '
> '--catalogd_args="--invalidate_tables_timeout_s=20" '
> 04:59:01 04:13:23 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 04:59:01 04:13:23 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 04:59:01 04:13:24 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 04:59:01 04:13:25 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 04:59:01 04:13:26 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 04:59:01 04:13:29 MainThread: Found 3 impalad/1 statestored/1 catalogd
> process(es)
> 04:59:01 04:13:29 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:29 MainThread: Waiting for num_known_live_backends=3. Current
> value: 0
> 04:59:01 04:13:30 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:30 MainThread: Waiting for num_known_live_backends=3. Current
> value: 1
> 04:59:01 04:13:31 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:31 MainThread: Waiting for num_known_live_backends=3. Current
> value: 2
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25001
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25002
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Impala Cluster Running with 3 nodes (3
> coordinators, 3 executors).
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25010
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 -- 2018-09-19
[jira] [Updated] (IMPALA-7593) test_automatic_invalidation failing in S3
[
https://issues.apache.org/jira/browse/IMPALA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-7593:
Component/s: Infrastructure
> test_automatic_invalidation failing in S3
> -
>
> Key: IMPALA-7593
> URL: https://issues.apache.org/jira/browse/IMPALA-7593
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> Note that the build has the fix for IMPALA-7580
> {noformat}
> 4:59:01 ___ TestAutomaticCatalogInvalidation.test_v1_catalog
> ___
> 04:59:01 custom_cluster/test_automatic_invalidation.py:63: in test_v1_catalog
> 04:59:01 self._run_test(cursor)
> 04:59:01 custom_cluster/test_automatic_invalidation.py:58: in _run_test
> 04:59:01 assert time.time() < timeout
> 04:59:01 E assert 1537355634.805718 < 1537355634.394429
> 04:59:01 E+ where 1537355634.805718 = ()
> 04:59:01 E+where = time.time
> 04:59:01 Captured stderr setup
> -
> 04:59:01 -- 2018-09-19 04:13:22,796 INFO MainThread: Starting cluster
> with command:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py
> --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args="--invalidate_tables_timeout_s=20" '
> '--state_store_args="--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50" '
> '--catalogd_args="--invalidate_tables_timeout_s=20" '
> 04:59:01 04:13:23 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 04:59:01 04:13:23 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 04:59:01 04:13:24 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 04:59:01 04:13:25 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 04:59:01 04:13:26 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 04:59:01 04:13:29 MainThread: Found 3 impalad/1 statestored/1 catalogd
> process(es)
> 04:59:01 04:13:29 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:29 MainThread: Waiting for num_known_live_backends=3. Current
> value: 0
> 04:59:01 04:13:30 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:30 MainThread: Waiting for num_known_live_backends=3. Current
> value: 1
> 04:59:01 04:13:31 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:31 MainThread: Waiting for num_known_live_backends=3. Current
> value: 2
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25001
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25002
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Impala Cluster Running with 3 nodes (3
> coordinators, 3 executors).
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25010
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Getting
> num_known_live_backends from
>
[jira] [Resolved] (IMPALA-7593) test_automatic_invalidation failing in S3
[
https://issues.apache.org/jira/browse/IMPALA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7593.
-
Resolution: Fixed
> test_automatic_invalidation failing in S3
> -
>
> Key: IMPALA-7593
> URL: https://issues.apache.org/jira/browse/IMPALA-7593
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build
>
> Note that the build has the fix for IMPALA-7580
> {noformat}
> 4:59:01 ___ TestAutomaticCatalogInvalidation.test_v1_catalog
> ___
> 04:59:01 custom_cluster/test_automatic_invalidation.py:63: in test_v1_catalog
> 04:59:01 self._run_test(cursor)
> 04:59:01 custom_cluster/test_automatic_invalidation.py:58: in _run_test
> 04:59:01 assert time.time() < timeout
> 04:59:01 E assert 1537355634.805718 < 1537355634.394429
> 04:59:01 E+ where 1537355634.805718 = ()
> 04:59:01 E+where = time.time
> 04:59:01 Captured stderr setup
> -
> 04:59:01 -- 2018-09-19 04:13:22,796 INFO MainThread: Starting cluster
> with command:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py
> --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args="--invalidate_tables_timeout_s=20" '
> '--state_store_args="--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50" '
> '--catalogd_args="--invalidate_tables_timeout_s=20" '
> 04:59:01 04:13:23 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 04:59:01 04:13:23 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 04:59:01 04:13:24 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 04:59:01 04:13:25 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 04:59:01 04:13:26 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 04:59:01 04:13:29 MainThread: Found 3 impalad/1 statestored/1 catalogd
> process(es)
> 04:59:01 04:13:29 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:29 MainThread: Waiting for num_known_live_backends=3. Current
> value: 0
> 04:59:01 04:13:30 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:30 MainThread: Waiting for num_known_live_backends=3. Current
> value: 1
> 04:59:01 04:13:31 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:31 MainThread: Waiting for num_known_live_backends=3. Current
> value: 2
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25001
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25002
> 04:59:01 04:13:32 MainThread: num_known_live_backends has reached value: 3
> 04:59:01 04:13:32 MainThread: Impala Cluster Running with 3 nodes (3
> coordinators, 3 executors).
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> 04:59:01 -- 2018-09-19 04:13:33,034 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25010
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> 04:59:01 -- 2018-09-19 04:13:33,036 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r4-4xlarge-ondemand-1860.vpc.cloudera.com:25000
> 04:59:01 -- 2018-09-19
[jira] [Comment Edited] (IMPALA-7606) Time based auto invalidation not working
[ https://issues.apache.org/jira/browse/IMPALA-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626623#comment-16626623 ] Tianyi Wang edited comment on IMPALA-7606 at 9/25/18 12:39 AM: --- [~parna] I ran impala tests for an hour with invalidate_tables_timeout_s = 60 and haven't seen this error. Can you provide a complete catalog log file? was (Author: tianyiwang): [~parna] I ran impala tests for an hour with invalidate_tables_timeout_s = 60 and hasn't seen this error. Can you provide a complete catalog log file? > Time based auto invalidation not working > > > Key: IMPALA-7606 > URL: https://issues.apache.org/jira/browse/IMPALA-7606 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Rituparna Agrawal >Assignee: Tianyi Wang >Priority: Major > > Set the invalidate_tables_timeout_s to 60 on both catalogd and all impalads. > Have tables populated. Catalogd is keeps throwing this exception. Tables are > not getting invalidated. > No other logs seen on impalad from CatalogTableInvalidator > [10:50 AM] Parna Agrawal: > [10:50 AM] Parna Agrawal: W0921 10:50:19.465893 13428 > CatalogdTableInvalidator.java:284] Unexpected exception thrown while > attempting to automatically invalidate tables. Will retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) > W0921 10:50:24.467514 13428 CatalogdTableInvalidator.java:284] Unexpected > exception thrown while attempting to automatically invalidate tables. Will > retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7606) Time based auto invalidation not working
[ https://issues.apache.org/jira/browse/IMPALA-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626623#comment-16626623 ] Tianyi Wang commented on IMPALA-7606: - [~parna] I ran impala tests for an hour with invalidate_tables_timeout_s = 60 and hasn't seen this error. Can you provide a complete catalog log file? > Time based auto invalidation not working > > > Key: IMPALA-7606 > URL: https://issues.apache.org/jira/browse/IMPALA-7606 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Rituparna Agrawal >Assignee: Tianyi Wang >Priority: Major > > Set the invalidate_tables_timeout_s to 60 on both catalogd and all impalads. > Have tables populated. Catalogd is keeps throwing this exception. Tables are > not getting invalidated. > No other logs seen on impalad from CatalogTableInvalidator > [10:50 AM] Parna Agrawal: > [10:50 AM] Parna Agrawal: W0921 10:50:19.465893 13428 > CatalogdTableInvalidator.java:284] Unexpected exception thrown while > attempting to automatically invalidate tables. Will retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) > W0921 10:50:24.467514 13428 CatalogdTableInvalidator.java:284] Unexpected > exception thrown while attempting to automatically invalidate tables. Will > retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7606) Time based auto invalidation not working
[ https://issues.apache.org/jira/browse/IMPALA-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang reassigned IMPALA-7606: --- Assignee: Tianyi Wang > Time based auto invalidation not working > > > Key: IMPALA-7606 > URL: https://issues.apache.org/jira/browse/IMPALA-7606 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 3.1.0 >Reporter: Rituparna Agrawal >Assignee: Tianyi Wang >Priority: Major > > Set the invalidate_tables_timeout_s to 60 on both catalogd and all impalads. > Have tables populated. Catalogd is keeps throwing this exception. Tables are > not getting invalidated. > No other logs seen on impalad from CatalogTableInvalidator > [10:50 AM] Parna Agrawal: > [10:50 AM] Parna Agrawal: W0921 10:50:19.465893 13428 > CatalogdTableInvalidator.java:284] Unexpected exception thrown while > attempting to automatically invalidate tables. Will retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) > W0921 10:50:24.467514 13428 CatalogdTableInvalidator.java:284] Unexpected > exception thrown while attempting to automatically invalidate tables. Will > retry in 5 seconds. > Java exception follows: > java.lang.IllegalStateException > at com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.impala.catalog.Table.getLastUsedTime(Table.java:174) > at > org.apache.impala.catalog.CatalogdTableInvalidator.invalidateOlderThan(CatalogdTableInvalidator.java:229) > at > org.apache.impala.catalog.CatalogdTableInvalidator.access$900(CatalogdTableInvalidator.java:51) > at > org.apache.impala.catalog.CatalogdTableInvalidator$DaemonThread.run(CatalogdTableInvalidator.java:275) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7532) Add retry/back-off to fetch-from-catalog RPCs
[ https://issues.apache.org/jira/browse/IMPALA-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16619890#comment-16619890 ] Tianyi Wang commented on IMPALA-7532: - [~vukercegovac] Do you know the details about the problem? AFAIK the main RPC used by fetch-from-catalogd is GetPartialCatalogObject, which has retrial mechanisms in catalog-op-executor.cc > Add retry/back-off to fetch-from-catalog RPCs > - > > Key: IMPALA-7532 > URL: https://issues.apache.org/jira/browse/IMPALA-7532 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Tianyi Wang >Priority: Major > > Currently if there is an error connecting to the catalog server, the 'fetch > from catalog' implementation will retry with no apparent backoff. We should > retry for some period of time with backoff in between the attempts, so that > impala can ride over short interruptions of the catalog service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7580) test_local_catalog fails on S3 build
[
https://issues.apache.org/jira/browse/IMPALA-7580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7580.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> test_local_catalog fails on S3 build
>
>
> Key: IMPALA-7580
> URL: https://issues.apache.org/jira/browse/IMPALA-7580
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> {noformat}
> 07:07:19 _ TestAutomaticCatalogInvalidation.test_local_catalog
> __
> 07:07:19 custom_cluster/test_automatic_invalidation.py:65: in
> test_local_catalog
> 07:07:19 self._run_test(cursor)
> 07:07:19 custom_cluster/test_automatic_invalidation.py:45: in _run_test
> 07:07:19 assert self.metadata_cache_string in self._get_catalog_object()
> 07:07:19 E assert 'columns (list) = liststruct' in ' 07:07:19 -- 2018-09-16 06:23:18,238 INFO MainThread: Starting cluster
> with command:
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py
> --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args="--invalidate_tables_timeout_s=5
> --use_local_catalog" '
> '--state_store_args="--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50" '
> '--catalogd_args="--invalidate_tables_timeout_s=5
> --catalog_topic_mode=minimal" '
> 07:07:19 06:23:18 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 07:07:19 06:23:18 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 07:07:19 06:23:20 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 07:07:19 06:23:21 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 07:07:19 06:23:22 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 07:07:19 06:23:25 MainThread: Found 3 impalad/1 statestored/1 catalogd
> process(es)
> 07:07:19 06:23:25 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25000
> 07:07:19 06:23:25 MainThread: Waiting for num_known_live_backends=3. Current
> value: 1
> 07:07:19 06:23:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25000
> 07:07:19 06:23:26 MainThread: num_known_live_backends has reached value: 3
> 07:07:19 06:23:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25001
> 07:07:19 06:23:26 MainThread: num_known_live_backends has reached value: 3
> 07:07:19 06:23:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25002
> 07:07:19 06:23:26 MainThread: num_known_live_backends has reached value: 3
> 07:07:19 06:23:26 MainThread: Impala Cluster Running with 3 nodes (3
> coordinators, 3 executors).
> 07:07:19 -- 2018-09-16 06:23:26,087 INFO MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> 07:07:19 -- 2018-09-16 06:23:26,088 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25010
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25000
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread:
> num_known_live_backends has reached value: 3
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25001
> 07:07:19 -- 2018-09-16 06:23:26,090 INFO MainThread:
> num_known_live_backends has reached value: 3
> 07:07:19 -- 2018-09-16 06:23:26,090 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25002
> 07:07:19 -- 2018-09-16 06:23:26,091 INFO MainThread:
>
[jira] [Resolved] (IMPALA-7580) test_local_catalog fails on S3 build
[
https://issues.apache.org/jira/browse/IMPALA-7580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7580.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> test_local_catalog fails on S3 build
>
>
> Key: IMPALA-7580
> URL: https://issues.apache.org/jira/browse/IMPALA-7580
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> {noformat}
> 07:07:19 _ TestAutomaticCatalogInvalidation.test_local_catalog
> __
> 07:07:19 custom_cluster/test_automatic_invalidation.py:65: in
> test_local_catalog
> 07:07:19 self._run_test(cursor)
> 07:07:19 custom_cluster/test_automatic_invalidation.py:45: in _run_test
> 07:07:19 assert self.metadata_cache_string in self._get_catalog_object()
> 07:07:19 E assert 'columns (list) = liststruct' in ' 07:07:19 -- 2018-09-16 06:23:18,238 INFO MainThread: Starting cluster
> with command:
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py
> --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args="--invalidate_tables_timeout_s=5
> --use_local_catalog" '
> '--state_store_args="--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50" '
> '--catalogd_args="--invalidate_tables_timeout_s=5
> --catalog_topic_mode=minimal" '
> 07:07:19 06:23:18 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 07:07:19 06:23:18 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 07:07:19 06:23:20 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 07:07:19 06:23:21 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 07:07:19 06:23:22 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 07:07:19 06:23:25 MainThread: Found 3 impalad/1 statestored/1 catalogd
> process(es)
> 07:07:19 06:23:25 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25000
> 07:07:19 06:23:25 MainThread: Waiting for num_known_live_backends=3. Current
> value: 1
> 07:07:19 06:23:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25000
> 07:07:19 06:23:26 MainThread: num_known_live_backends has reached value: 3
> 07:07:19 06:23:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25001
> 07:07:19 06:23:26 MainThread: num_known_live_backends has reached value: 3
> 07:07:19 06:23:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25002
> 07:07:19 06:23:26 MainThread: num_known_live_backends has reached value: 3
> 07:07:19 06:23:26 MainThread: Impala Cluster Running with 3 nodes (3
> coordinators, 3 executors).
> 07:07:19 -- 2018-09-16 06:23:26,087 INFO MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> 07:07:19 -- 2018-09-16 06:23:26,088 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25010
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25000
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread:
> num_known_live_backends has reached value: 3
> 07:07:19 -- 2018-09-16 06:23:26,089 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25001
> 07:07:19 -- 2018-09-16 06:23:26,090 INFO MainThread:
> num_known_live_backends has reached value: 3
> 07:07:19 -- 2018-09-16 06:23:26,090 INFO MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-0c64.vpc.cloudera.com:25002
> 07:07:19 -- 2018-09-16 06:23:26,091 INFO MainThread:
>
[jira] [Created] (IMPALA-7448) Periodically evict recently unused table from catalogd
Tianyi Wang created IMPALA-7448: --- Summary: Periodically evict recently unused table from catalogd Key: IMPALA-7448 URL: https://issues.apache.org/jira/browse/IMPALA-7448 Project: IMPALA Issue Type: Improvement Components: Catalog Affects Versions: Impala 3.1.0 Reporter: Tianyi Wang Assignee: Tianyi Wang To limit the memory consumption of catalog, we should experiment with a mechanism automatically evicting recently unused tables from catalogd. Initial design: - impalad to report periodically/asynchronously the set of catalog objects that were accessed - catalogd to record some kind of last access time - catalogd to have some facility to scan over all catalog objects, collect some number of not-recently-used ones (eg to reach a target amount of evicted memory), and issue invalidate commands to itself - no need to have exact LRU behavior -- to simplify, we probably shouldn't try to do a classical LRU linked list between all catalog objects. - initial patch probably just triggered manually. Discussed either running this on a schedule or running this based on JMX GC notifications if we see that the catalogd finished an old-gen GC and the old gen is more than some target percentage full. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7448) Periodically evict recently unused table from catalogd
Tianyi Wang created IMPALA-7448: --- Summary: Periodically evict recently unused table from catalogd Key: IMPALA-7448 URL: https://issues.apache.org/jira/browse/IMPALA-7448 Project: IMPALA Issue Type: Improvement Components: Catalog Affects Versions: Impala 3.1.0 Reporter: Tianyi Wang Assignee: Tianyi Wang To limit the memory consumption of catalog, we should experiment with a mechanism automatically evicting recently unused tables from catalogd. Initial design: - impalad to report periodically/asynchronously the set of catalog objects that were accessed - catalogd to record some kind of last access time - catalogd to have some facility to scan over all catalog objects, collect some number of not-recently-used ones (eg to reach a target amount of evicted memory), and issue invalidate commands to itself - no need to have exact LRU behavior -- to simplify, we probably shouldn't try to do a classical LRU linked list between all catalog objects. - initial patch probably just triggered manually. Discussed either running this on a schedule or running this based on JMX GC notifications if we see that the catalogd finished an old-gen GC and the old gen is more than some target percentage full. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7347) Assertion Failure - test_show_create_table
[
https://issues.apache.org/jira/browse/IMPALA-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7347.
-
Resolution: Fixed
> Assertion Failure - test_show_create_table
> ---
>
> Key: IMPALA-7347
> URL: https://issues.apache.org/jira/browse/IMPALA-7347
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: build-failure
> Fix For: Impala 3.1.0
>
>
> test_show_create_table in metadata/test_show_create_table.py is failing with
> the following assertion error
> {noformat}
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database)
> metadata/test_show_create_table.py:106: in __run_show_create_table_test_case
> self.__compare_result(expected_result, create_table_result)
> metadata/test_show_create_table.py:134: in __compare_result
> assert expected_tbl_props == actual_tbl_props
> E assert {} == {'numFilesErasureCoded': '0'}
> E Right contains more items:
> E {'numFilesErasureCoded': '0'}
> E Use -v to get the full diff
> {noformat}
>
> It appears that table property "numFilesErasureCoded" is showing up in table
> properties.
> Either the test needs updating or a bug.
>
> {code}
> h3. Error Message
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database) metadata/test_show_create_table.py:106: in
> __run_show_create_table_test_case self.__compare_result(expected_result,
> create_table_result) metadata/test_show_create_table.py:134: in
> __compare_result assert expected_tbl_props == actual_tbl_props E assert {} ==
> \{'numFilesErasureCoded': '0'} E Right contains more items: E
> \{'numFilesErasureCoded': '0'} E Use -v to get the full diff
> {code}
>
> ---
> {code}
> h3. Standard Error
> -- connecting to: localhost:21000 SET sync_ddl=False; -- executing against
> localhost:21000 DROP DATABASE IF EXISTS `test_show_create_table_f1598d0b`
> CASCADE; SET sync_ddl=False; -- executing against localhost:21000 CREATE
> DATABASE `test_show_create_table_f1598d0b`; MainThread: Created database
> "test_show_create_table_f1598d0b" for test ID
> "metadata/test_show_create_table.py::TestShowCreateTable::()::test_show_create_table[table_format:
> text/none]" -- executing against localhost:21000 CREATE TABLE
> test_show_create_table_f1598d0b.test1 ( id INT ) STORED AS TEXTFILE; --
> executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test1 (id INT)
> STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test1';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test2 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) STORED AS TEXTFILE; -- executing against
> localhost:21000 show create table test_show_create_table_f1598d0b.test2; --
> executing against localhost:21000 drop table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> CREATE TABLE test_show_create_table_f1598d0b.test2 (year INT, month INT, id
> INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col TINYINT,
> smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col FLOAT,
> double_col DOUBLE, date_string_col STRING, string_col STRING, timestamp_col
> TIMESTAMP) STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test2';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test2; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test3 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) PARTITIONED BY ( x INT, y INT, a BOOLEAN ) COMMENT
> 'This is a test' STORED AS TEXTFILE; -- executing against localhost:21000
> show create table
[jira] [Resolved] (IMPALA-7347) Assertion Failure - test_show_create_table
[
https://issues.apache.org/jira/browse/IMPALA-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7347.
-
Resolution: Fixed
> Assertion Failure - test_show_create_table
> ---
>
> Key: IMPALA-7347
> URL: https://issues.apache.org/jira/browse/IMPALA-7347
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: build-failure
> Fix For: Impala 3.1.0
>
>
> test_show_create_table in metadata/test_show_create_table.py is failing with
> the following assertion error
> {noformat}
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database)
> metadata/test_show_create_table.py:106: in __run_show_create_table_test_case
> self.__compare_result(expected_result, create_table_result)
> metadata/test_show_create_table.py:134: in __compare_result
> assert expected_tbl_props == actual_tbl_props
> E assert {} == {'numFilesErasureCoded': '0'}
> E Right contains more items:
> E {'numFilesErasureCoded': '0'}
> E Use -v to get the full diff
> {noformat}
>
> It appears that table property "numFilesErasureCoded" is showing up in table
> properties.
> Either the test needs updating or a bug.
>
> {code}
> h3. Error Message
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database) metadata/test_show_create_table.py:106: in
> __run_show_create_table_test_case self.__compare_result(expected_result,
> create_table_result) metadata/test_show_create_table.py:134: in
> __compare_result assert expected_tbl_props == actual_tbl_props E assert {} ==
> \{'numFilesErasureCoded': '0'} E Right contains more items: E
> \{'numFilesErasureCoded': '0'} E Use -v to get the full diff
> {code}
>
> ---
> {code}
> h3. Standard Error
> -- connecting to: localhost:21000 SET sync_ddl=False; -- executing against
> localhost:21000 DROP DATABASE IF EXISTS `test_show_create_table_f1598d0b`
> CASCADE; SET sync_ddl=False; -- executing against localhost:21000 CREATE
> DATABASE `test_show_create_table_f1598d0b`; MainThread: Created database
> "test_show_create_table_f1598d0b" for test ID
> "metadata/test_show_create_table.py::TestShowCreateTable::()::test_show_create_table[table_format:
> text/none]" -- executing against localhost:21000 CREATE TABLE
> test_show_create_table_f1598d0b.test1 ( id INT ) STORED AS TEXTFILE; --
> executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test1 (id INT)
> STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test1';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test2 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) STORED AS TEXTFILE; -- executing against
> localhost:21000 show create table test_show_create_table_f1598d0b.test2; --
> executing against localhost:21000 drop table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> CREATE TABLE test_show_create_table_f1598d0b.test2 (year INT, month INT, id
> INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col TINYINT,
> smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col FLOAT,
> double_col DOUBLE, date_string_col STRING, string_col STRING, timestamp_col
> TIMESTAMP) STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test2';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test2; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test3 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) PARTITIONED BY ( x INT, y INT, a BOOLEAN ) COMMENT
> 'This is a test' STORED AS TEXTFILE; -- executing against localhost:21000
> show create table
[jira] [Resolved] (IMPALA-7383) Make METASTORE_DB configurable and default to escaped $IMPALA_HOME
[ https://issues.apache.org/jira/browse/IMPALA-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7383. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Make METASTORE_DB configurable and default to escaped $IMPALA_HOME > -- > > Key: IMPALA-7383 > URL: https://issues.apache.org/jira/browse/IMPALA-7383 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Trivial > Labels: infrastructure > Fix For: Impala 3.1.0 > > > Some developers keep multiple impala repos on their disk. Isolating > METASTORE_DB may help with switching between those repos without needing to > reload the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7383) Make METASTORE_DB configurable and default to escaped $IMPALA_HOME
[ https://issues.apache.org/jira/browse/IMPALA-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7383. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Make METASTORE_DB configurable and default to escaped $IMPALA_HOME > -- > > Key: IMPALA-7383 > URL: https://issues.apache.org/jira/browse/IMPALA-7383 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Assignee: Tianyi Wang >Priority: Trivial > Labels: infrastructure > Fix For: Impala 3.1.0 > > > Some developers keep multiple impala repos on their disk. Isolating > METASTORE_DB may help with switching between those repos without needing to > reload the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (IMPALA-7347) Assertion Failure - test_show_create_table
[
https://issues.apache.org/jira/browse/IMPALA-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang reopened IMPALA-7347:
-
The fix didn't consider the case where the FS is local.
> Assertion Failure - test_show_create_table
> ---
>
> Key: IMPALA-7347
> URL: https://issues.apache.org/jira/browse/IMPALA-7347
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: build-failure
> Fix For: Impala 3.1.0
>
>
> test_show_create_table in metadata/test_show_create_table.py is failing with
> the following assertion error
> {noformat}
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database)
> metadata/test_show_create_table.py:106: in __run_show_create_table_test_case
> self.__compare_result(expected_result, create_table_result)
> metadata/test_show_create_table.py:134: in __compare_result
> assert expected_tbl_props == actual_tbl_props
> E assert {} == {'numFilesErasureCoded': '0'}
> E Right contains more items:
> E {'numFilesErasureCoded': '0'}
> E Use -v to get the full diff
> {noformat}
>
> It appears that table property "numFilesErasureCoded" is showing up in table
> properties.
> Either the test needs updating or a bug.
>
> {code}
> h3. Error Message
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database) metadata/test_show_create_table.py:106: in
> __run_show_create_table_test_case self.__compare_result(expected_result,
> create_table_result) metadata/test_show_create_table.py:134: in
> __compare_result assert expected_tbl_props == actual_tbl_props E assert {} ==
> \{'numFilesErasureCoded': '0'} E Right contains more items: E
> \{'numFilesErasureCoded': '0'} E Use -v to get the full diff
> {code}
>
> ---
> {code}
> h3. Standard Error
> -- connecting to: localhost:21000 SET sync_ddl=False; -- executing against
> localhost:21000 DROP DATABASE IF EXISTS `test_show_create_table_f1598d0b`
> CASCADE; SET sync_ddl=False; -- executing against localhost:21000 CREATE
> DATABASE `test_show_create_table_f1598d0b`; MainThread: Created database
> "test_show_create_table_f1598d0b" for test ID
> "metadata/test_show_create_table.py::TestShowCreateTable::()::test_show_create_table[table_format:
> text/none]" -- executing against localhost:21000 CREATE TABLE
> test_show_create_table_f1598d0b.test1 ( id INT ) STORED AS TEXTFILE; --
> executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test1 (id INT)
> STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test1';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test2 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) STORED AS TEXTFILE; -- executing against
> localhost:21000 show create table test_show_create_table_f1598d0b.test2; --
> executing against localhost:21000 drop table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> CREATE TABLE test_show_create_table_f1598d0b.test2 (year INT, month INT, id
> INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col TINYINT,
> smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col FLOAT,
> double_col DOUBLE, date_string_col STRING, string_col STRING, timestamp_col
> TIMESTAMP) STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test2';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test2; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test3 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) PARTITIONED BY ( x INT, y INT, a BOOLEAN ) COMMENT
> 'This is a test' STORED AS TEXTFILE; -- executing against localhost:21000
> show create table
[jira] [Resolved] (IMPALA-7347) Assertion Failure - test_show_create_table
[
https://issues.apache.org/jira/browse/IMPALA-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7347.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Assertion Failure - test_show_create_table
> ---
>
> Key: IMPALA-7347
> URL: https://issues.apache.org/jira/browse/IMPALA-7347
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: build-failure
> Fix For: Impala 3.1.0
>
>
> test_show_create_table in metadata/test_show_create_table.py is failing with
> the following assertion error
> {noformat}
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database)
> metadata/test_show_create_table.py:106: in __run_show_create_table_test_case
> self.__compare_result(expected_result, create_table_result)
> metadata/test_show_create_table.py:134: in __compare_result
> assert expected_tbl_props == actual_tbl_props
> E assert {} == {'numFilesErasureCoded': '0'}
> E Right contains more items:
> E {'numFilesErasureCoded': '0'}
> E Use -v to get the full diff
> {noformat}
>
> It appears that table property "numFilesErasureCoded" is showing up in table
> properties.
> Either the test needs updating or a bug.
>
> {code}
> h3. Error Message
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database) metadata/test_show_create_table.py:106: in
> __run_show_create_table_test_case self.__compare_result(expected_result,
> create_table_result) metadata/test_show_create_table.py:134: in
> __compare_result assert expected_tbl_props == actual_tbl_props E assert {} ==
> \{'numFilesErasureCoded': '0'} E Right contains more items: E
> \{'numFilesErasureCoded': '0'} E Use -v to get the full diff
> {code}
>
> ---
> {code}
> h3. Standard Error
> -- connecting to: localhost:21000 SET sync_ddl=False; -- executing against
> localhost:21000 DROP DATABASE IF EXISTS `test_show_create_table_f1598d0b`
> CASCADE; SET sync_ddl=False; -- executing against localhost:21000 CREATE
> DATABASE `test_show_create_table_f1598d0b`; MainThread: Created database
> "test_show_create_table_f1598d0b" for test ID
> "metadata/test_show_create_table.py::TestShowCreateTable::()::test_show_create_table[table_format:
> text/none]" -- executing against localhost:21000 CREATE TABLE
> test_show_create_table_f1598d0b.test1 ( id INT ) STORED AS TEXTFILE; --
> executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test1 (id INT)
> STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test1';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test2 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) STORED AS TEXTFILE; -- executing against
> localhost:21000 show create table test_show_create_table_f1598d0b.test2; --
> executing against localhost:21000 drop table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> CREATE TABLE test_show_create_table_f1598d0b.test2 (year INT, month INT, id
> INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col TINYINT,
> smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col FLOAT,
> double_col DOUBLE, date_string_col STRING, string_col STRING, timestamp_col
> TIMESTAMP) STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test2';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test2; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test3 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) PARTITIONED BY ( x INT, y INT, a BOOLEAN ) COMMENT
> 'This is a test' STORED AS TEXTFILE; -- executing against localhost:21000
> show create table
[jira] [Resolved] (IMPALA-7347) Assertion Failure - test_show_create_table
[
https://issues.apache.org/jira/browse/IMPALA-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7347.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Assertion Failure - test_show_create_table
> ---
>
> Key: IMPALA-7347
> URL: https://issues.apache.org/jira/browse/IMPALA-7347
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: nithya
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: build-failure
> Fix For: Impala 3.1.0
>
>
> test_show_create_table in metadata/test_show_create_table.py is failing with
> the following assertion error
> {noformat}
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database)
> metadata/test_show_create_table.py:106: in __run_show_create_table_test_case
> self.__compare_result(expected_result, create_table_result)
> metadata/test_show_create_table.py:134: in __compare_result
> assert expected_tbl_props == actual_tbl_props
> E assert {} == {'numFilesErasureCoded': '0'}
> E Right contains more items:
> E {'numFilesErasureCoded': '0'}
> E Use -v to get the full diff
> {noformat}
>
> It appears that table property "numFilesErasureCoded" is showing up in table
> properties.
> Either the test needs updating or a bug.
>
> {code}
> h3. Error Message
> metadata/test_show_create_table.py:58: in test_show_create_table
> unique_database) metadata/test_show_create_table.py:106: in
> __run_show_create_table_test_case self.__compare_result(expected_result,
> create_table_result) metadata/test_show_create_table.py:134: in
> __compare_result assert expected_tbl_props == actual_tbl_props E assert {} ==
> \{'numFilesErasureCoded': '0'} E Right contains more items: E
> \{'numFilesErasureCoded': '0'} E Use -v to get the full diff
> {code}
>
> ---
> {code}
> h3. Standard Error
> -- connecting to: localhost:21000 SET sync_ddl=False; -- executing against
> localhost:21000 DROP DATABASE IF EXISTS `test_show_create_table_f1598d0b`
> CASCADE; SET sync_ddl=False; -- executing against localhost:21000 CREATE
> DATABASE `test_show_create_table_f1598d0b`; MainThread: Created database
> "test_show_create_table_f1598d0b" for test ID
> "metadata/test_show_create_table.py::TestShowCreateTable::()::test_show_create_table[table_format:
> text/none]" -- executing against localhost:21000 CREATE TABLE
> test_show_create_table_f1598d0b.test1 ( id INT ) STORED AS TEXTFILE; --
> executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test1 (id INT)
> STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test1';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test1; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test1; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test2 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) STORED AS TEXTFILE; -- executing against
> localhost:21000 show create table test_show_create_table_f1598d0b.test2; --
> executing against localhost:21000 drop table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> CREATE TABLE test_show_create_table_f1598d0b.test2 (year INT, month INT, id
> INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col TINYINT,
> smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col FLOAT,
> double_col DOUBLE, date_string_col STRING, string_col STRING, timestamp_col
> TIMESTAMP) STORED AS TEXTFILE LOCATION
> 'hdfs://localhost:20500/test-warehouse/test_show_create_table_f1598d0b.db/test2';
> -- executing against localhost:21000 show create table
> test_show_create_table_f1598d0b.test2; -- executing against localhost:21000
> drop table test_show_create_table_f1598d0b.test2; -- executing against
> localhost:21000 CREATE TABLE test_show_create_table_f1598d0b.test3 ( year
> INT, month INT, id INT COMMENT 'Add a comment', bool_col BOOLEAN, tinyint_col
> TINYINT, smallint_col SMALLINT, int_col INT, bigint_col BIGINT, float_col
> FLOAT, double_col DOUBLE, date_string_col STRING, string_col STRING,
> timestamp_col TIMESTAMP ) PARTITIONED BY ( x INT, y INT, a BOOLEAN ) COMMENT
> 'This is a test' STORED AS TEXTFILE; -- executing against localhost:21000
> show create table
[jira] [Created] (IMPALA-7384) Move /var/lib/hadoop-hdfs into IMPALA_HOME
Tianyi Wang created IMPALA-7384: --- Summary: Move /var/lib/hadoop-hdfs into IMPALA_HOME Key: IMPALA-7384 URL: https://issues.apache.org/jira/browse/IMPALA-7384 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Tianyi Wang Assignee: Tianyi Wang Currently the minicluster uses /var/lib/hadoop-hdfs as HDFS shortcut socket dir. It's a hard-coded path, requires root privilege to create, is a leaky resource, and is one of the many reasons preventing developers to run multiple mini-clusters concurrently. We should put it into the testdata/cluster directory instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7384) Move /var/lib/hadoop-hdfs into IMPALA_HOME
Tianyi Wang created IMPALA-7384: --- Summary: Move /var/lib/hadoop-hdfs into IMPALA_HOME Key: IMPALA-7384 URL: https://issues.apache.org/jira/browse/IMPALA-7384 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Tianyi Wang Assignee: Tianyi Wang Currently the minicluster uses /var/lib/hadoop-hdfs as HDFS shortcut socket dir. It's a hard-coded path, requires root privilege to create, is a leaky resource, and is one of the many reasons preventing developers to run multiple mini-clusters concurrently. We should put it into the testdata/cluster directory instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7383) Make METASTORE_DB configurable and default to escaped $IMPALA_HOME
Tianyi Wang created IMPALA-7383: --- Summary: Make METASTORE_DB configurable and default to escaped $IMPALA_HOME Key: IMPALA-7383 URL: https://issues.apache.org/jira/browse/IMPALA-7383 Project: IMPALA Issue Type: Improvement Affects Versions: Impala 3.1.0 Reporter: Tianyi Wang Assignee: Tianyi Wang Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB may help with switching between those repos without needing to reload the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7383) Make METASTORE_DB configurable and default to escaped $IMPALA_HOME
Tianyi Wang created IMPALA-7383: --- Summary: Make METASTORE_DB configurable and default to escaped $IMPALA_HOME Key: IMPALA-7383 URL: https://issues.apache.org/jira/browse/IMPALA-7383 Project: IMPALA Issue Type: Improvement Affects Versions: Impala 3.1.0 Reporter: Tianyi Wang Assignee: Tianyi Wang Some developers keep multiple impala repos on their disk. Isolating METASTORE_DB may help with switching between those repos without needing to reload the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7252) Backport rate limiting of fadvise calls into toolchain glog
[ https://issues.apache.org/jira/browse/IMPALA-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7252. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Backport rate limiting of fadvise calls into toolchain glog > --- > > Key: IMPALA-7252 > URL: https://issues.apache.org/jira/browse/IMPALA-7252 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0 >Reporter: Todd Lipcon >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > > Currently, glog's default behavior is to call fadvise(FADV_DONTNEED) on the > log file after each entry that is written. In many versions of the Linux > kernel, each invocation of this call causes work to be scheduled on all other > CPUs, causing up to one context switch per CPU for every log line. We saw > this cause an extremely long GC pause in the catalogd in the case where the > native side of the catalog was logging a lot of messages about publishing > metadata updates at the same time that the Java side was running a GC. The GC > spent almost all of its time in the kernel due to the high context switch > rate causing a lot of TLB clears and misses, and instead of pausing the JVM > for a couple of seconds took several minutes. > This was identified and fixed upstream in glog here: > https://github.com/google/glog/commit/dacd29679633c9b845708e7015bd2c79367a6ea2 > We should backport this fix into the version in the toolchain. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7252) Backport rate limiting of fadvise calls into toolchain glog
[ https://issues.apache.org/jira/browse/IMPALA-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang resolved IMPALA-7252. - Resolution: Fixed Fix Version/s: Impala 3.1.0 > Backport rate limiting of fadvise calls into toolchain glog > --- > > Key: IMPALA-7252 > URL: https://issues.apache.org/jira/browse/IMPALA-7252 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0 >Reporter: Todd Lipcon >Assignee: Tianyi Wang >Priority: Major > Fix For: Impala 3.1.0 > > > Currently, glog's default behavior is to call fadvise(FADV_DONTNEED) on the > log file after each entry that is written. In many versions of the Linux > kernel, each invocation of this call causes work to be scheduled on all other > CPUs, causing up to one context switch per CPU for every log line. We saw > this cause an extremely long GC pause in the catalogd in the case where the > native side of the catalog was logging a lot of messages about publishing > metadata updates at the same time that the Java side was running a GC. The GC > spent almost all of its time in the kernel due to the high context switch > rate causing a lot of TLB clears and misses, and instead of pausing the JVM > for a couple of seconds took several minutes. > This was identified and fixed upstream in glog here: > https://github.com/google/glog/commit/dacd29679633c9b845708e7015bd2c79367a6ea2 > We should backport this fix into the version in the toolchain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7154) Error making 'dropDatabase' RPC to Hive Metastore
[
https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535512#comment-16535512
]
Tianyi Wang commented on IMPALA-7154:
-
- The other issues happen on non-S3 as well.
-
https://github.com/apache/impala/blob/fd0ba0fd2c7631d48b4cce56e60f0b9f902cc446/tests/run-tests.py#L51
> Error making 'dropDatabase' RPC to Hive Metastore
> -
>
> Key: IMPALA-7154
> URL: https://issues.apache.org/jira/browse/IMPALA-7154
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: TEST-impala-parallel.log.gz,
> TEST-impala-parallel.xml.gz,
> catalogd.ec2-m2-4xlarge-centos-6-4-0f46.vpc.cloudera.com.jenkins.log.INFO.20180608-024815.32143.gz,
> hive.log.gz
>
>
> {noformat}
> conftest.py:293: in cleanup
> {'sync_ddl': sync_ddl})
> common/impala_test_suite.py:528: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:535: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive
> Metastore:
> E CAUSED BY: NoSuchObjectException: test_resolution_by_name_56b45511
> {noformat}
> The backtrace in the catalogd log is:
> {noformat}
> I0608 05:49:26.111824 24195 jni-util.cc:230]
> org.apache.impala.common.ImpalaRuntimeException: Error making 'dropDatabase'
> RPC to Hive Metastore:
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: NoSuchObjectException(message:test_resolution_by_name_56b45511)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16387)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16364)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMetastore.java:16295)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:702)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:689)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7154) Error making 'dropDatabase' RPC to Hive Metastore
[
https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535501#comment-16535501
]
Tianyi Wang commented on IMPALA-7154:
-
We have quite a few JIRAs caused by lost connection to namenode/HMS or even
between impalads. I think we could try reducing the concurrency of the parallel
tests and see how it goes.
> Error making 'dropDatabase' RPC to Hive Metastore
> -
>
> Key: IMPALA-7154
> URL: https://issues.apache.org/jira/browse/IMPALA-7154
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: TEST-impala-parallel.log.gz,
> TEST-impala-parallel.xml.gz,
> catalogd.ec2-m2-4xlarge-centos-6-4-0f46.vpc.cloudera.com.jenkins.log.INFO.20180608-024815.32143.gz,
> hive.log.gz
>
>
> {noformat}
> conftest.py:293: in cleanup
> {'sync_ddl': sync_ddl})
> common/impala_test_suite.py:528: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:535: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive
> Metastore:
> E CAUSED BY: NoSuchObjectException: test_resolution_by_name_56b45511
> {noformat}
> The backtrace in the catalogd log is:
> {noformat}
> I0608 05:49:26.111824 24195 jni-util.cc:230]
> org.apache.impala.common.ImpalaRuntimeException: Error making 'dropDatabase'
> RPC to Hive Metastore:
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: NoSuchObjectException(message:test_resolution_by_name_56b45511)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16387)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16364)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMetastore.java:16295)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:702)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:689)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7236) Erasure coding dataload broken by IMPALA-7102
[
https://issues.apache.org/jira/browse/IMPALA-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7236.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Erasure coding dataload broken by IMPALA-7102
> -
>
> Key: IMPALA-7236
> URL: https://issues.apache.org/jira/browse/IMPALA-7236
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> Loading nested data now fails on erasure coding:
> {code:java}
> 16:58:45 Loading nested data (logging to
> /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/data_loading/load-nested.log)...
>
> 17:01:22 FAILED (Took: 2 min 37 sec)
> 17:01:22
> '/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/testdata/bin/load_nested.py'
> failed. Tail of log:
> 17:01:22 2018-06-30 16:59:54,241 INFO:load_nested[277]:Executing:
> 17:01:22
> 17:01:22 CREATE TABLE customer
> 17:01:22 STORED AS PARQUET
> 17:01:22 TBLPROPERTIES('parquet.compression'='SNAPPY')
> 17:01:22 AS SELECT * FROM tmp_customer
> 17:01:22 2018-06-30 17:01:00,454 INFO:load_nested[277]:Executing:
> 17:01:22
> 17:01:22 CREATE TABLE region
> 17:01:22 STORED AS PARQUET
> 17:01:22 TBLPROPERTIES('parquet.compression'='SNAPPY')
> 17:01:22 AS SELECT * FROM tmp_region
> 17:01:22 2018-06-30 17:01:01,744 INFO:load_nested[277]:Executing:
> 17:01:22
> 17:01:22 CREATE TABLE supplier
> 17:01:22 STORED AS PARQUET
> 17:01:22 TBLPROPERTIES('parquet.compression'='SNAPPY')
> 17:01:22 AS SELECT * FROM tmp_supplier
> 17:01:22 2018-06-30 17:01:06,033 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_orders_string
> 17:01:22 2018-06-30 17:01:06,097 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_customer_string
> 17:01:22 2018-06-30 17:01:06,127 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_customer
> 17:01:22 2018-06-30 17:01:10,067 INFO:load_nested[296]:Executing:
> 17:01:22
> 17:01:22 DROP TABLE tmp_region_string
> 17:01:22 2018-06-30 17:01:10,097 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_region
> 17:01:22 2018-06-30 17:01:14,071 INFO:load_nested[296]:Executing:
> 17:01:22
> 17:01:22 DROP TABLE tmp_supplier
> 17:01:22 2018-06-30 17:01:18,075 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_supplier_string
> 17:01:22 Traceback (most recent call last):
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/testdata/bin/load_nested.py",
> line 331, in
> 17:01:22 load()
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/testdata/bin/load_nested.py",
> line 303, in load
> 17:01:22 impala.compute_stats()
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/tests/comparison/db_connection.py",
> line 760, in compute_stats
> 17:01:22 self.execute("COMPUTE STATS %s" % table_name)
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/tests/comparison/db_connection.py",
> line 206, in execute
> 17:01:22 return self._cursor.execute(sql, *args, **kwargs)
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 304, in execute
> 17:01:22 self._wait_to_finish() # make execute synchronous
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 380, in _wait_to_finish
> 17:01:22 raise OperationalError(resp.errorMessage)
> 17:01:22 impala.error.OperationalError: ImpalaRuntimeException: Scanning of
> HDFS erasure-coded file
> (hdfs://localhost:20500/test-warehouse/tpch_nested_parquet.db/customer/00_0)
> is not supported
> 17:01:22
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7236) Erasure coding dataload broken by IMPALA-7102
[
https://issues.apache.org/jira/browse/IMPALA-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7236.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Erasure coding dataload broken by IMPALA-7102
> -
>
> Key: IMPALA-7236
> URL: https://issues.apache.org/jira/browse/IMPALA-7236
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> Loading nested data now fails on erasure coding:
> {code:java}
> 16:58:45 Loading nested data (logging to
> /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/data_loading/load-nested.log)...
>
> 17:01:22 FAILED (Took: 2 min 37 sec)
> 17:01:22
> '/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/testdata/bin/load_nested.py'
> failed. Tail of log:
> 17:01:22 2018-06-30 16:59:54,241 INFO:load_nested[277]:Executing:
> 17:01:22
> 17:01:22 CREATE TABLE customer
> 17:01:22 STORED AS PARQUET
> 17:01:22 TBLPROPERTIES('parquet.compression'='SNAPPY')
> 17:01:22 AS SELECT * FROM tmp_customer
> 17:01:22 2018-06-30 17:01:00,454 INFO:load_nested[277]:Executing:
> 17:01:22
> 17:01:22 CREATE TABLE region
> 17:01:22 STORED AS PARQUET
> 17:01:22 TBLPROPERTIES('parquet.compression'='SNAPPY')
> 17:01:22 AS SELECT * FROM tmp_region
> 17:01:22 2018-06-30 17:01:01,744 INFO:load_nested[277]:Executing:
> 17:01:22
> 17:01:22 CREATE TABLE supplier
> 17:01:22 STORED AS PARQUET
> 17:01:22 TBLPROPERTIES('parquet.compression'='SNAPPY')
> 17:01:22 AS SELECT * FROM tmp_supplier
> 17:01:22 2018-06-30 17:01:06,033 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_orders_string
> 17:01:22 2018-06-30 17:01:06,097 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_customer_string
> 17:01:22 2018-06-30 17:01:06,127 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_customer
> 17:01:22 2018-06-30 17:01:10,067 INFO:load_nested[296]:Executing:
> 17:01:22
> 17:01:22 DROP TABLE tmp_region_string
> 17:01:22 2018-06-30 17:01:10,097 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_region
> 17:01:22 2018-06-30 17:01:14,071 INFO:load_nested[296]:Executing:
> 17:01:22
> 17:01:22 DROP TABLE tmp_supplier
> 17:01:22 2018-06-30 17:01:18,075 INFO:load_nested[296]:Executing:
> 17:01:22 DROP TABLE tmp_supplier_string
> 17:01:22 Traceback (most recent call last):
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/testdata/bin/load_nested.py",
> line 331, in
> 17:01:22 load()
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/testdata/bin/load_nested.py",
> line 303, in load
> 17:01:22 impala.compute_stats()
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/tests/comparison/db_connection.py",
> line 760, in compute_stats
> 17:01:22 self.execute("COMPUTE STATS %s" % table_name)
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/tests/comparison/db_connection.py",
> line 206, in execute
> 17:01:22 return self._cursor.execute(sql, *args, **kwargs)
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 304, in execute
> 17:01:22 self._wait_to_finish() # make execute synchronous
> 17:01:22 File
> "/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 380, in _wait_to_finish
> 17:01:22 raise OperationalError(resp.errorMessage)
> 17:01:22 impala.error.OperationalError: ImpalaRuntimeException: Scanning of
> HDFS erasure-coded file
> (hdfs://localhost:20500/test-warehouse/tpch_nested_parquet.db/customer/00_0)
> is not supported
> 17:01:22
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7169.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
HDFS trash checkpointing renames files in the trash folder and breaks
impala tests. Impala set the trash checkpointing interval to 1440 to try
to postpone it for 24 hours. Unfortunately that told HDFS to do it when
the UNIX time is a multiple of 1440 * 60 and it broke trash-related
tests run around midnight in GMT. This patch sets the interval to
541728000 so that HDFS won't do the checkpointing until Jan 1st 3000,
and HDFS will checkpoint every 1030 years after that.
Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Reviewed-on: http://gerrit.cloudera.org:8080/10742
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7169.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
IMPALA-7169: Prevent HDFS from checkpointing trash until 3000 AD
HDFS trash checkpointing renames files in the trash folder and breaks
impala tests. Impala set the trash checkpointing interval to 1440 to try
to postpone it for 24 hours. Unfortunately that told HDFS to do it when
the UNIX time is a multiple of 1440 * 60 and it broke trash-related
tests run around midnight in GMT. This patch sets the interval to
541728000 so that HDFS won't do the checkpointing until Jan 1st 3000,
and HDFS will checkpoint every 1030 years after that.
Change-Id: I9452f7e44c7679f86a947cd20115c078757223d8
Reviewed-on: http://gerrit.cloudera.org:8080/10742
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7193) Local filesystem failes with fs.defaultFS (file:/tmp) is not supported
Tianyi Wang created IMPALA-7193:
---
Summary: Local filesystem failes with fs.defaultFS (file:/tmp) is
not supported
Key: IMPALA-7193
URL: https://issues.apache.org/jira/browse/IMPALA-7193
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Joe McDonnell
Impala failed to start on a local file system
{noformat}
I0619 07:25:34.034348 13157 status.cc:125] Currently configured default
filesystem: ProxyLocalFileSystem. fs.defaultFS (file:/tmp) is not supported.
@ 0x18a01f9 impala::Status::Status()
@ 0x1dd30b6 impala::Frontend::ValidateSettings()
@ 0x1dee2aa impala::ImpalaServer::ImpalaServer()
@ 0x1dea8db ImpaladMain()
@ 0x185c380 main
@ 0x7f373f829c05 __libc_start_main
@ 0x185c1f1 (unknown)
E0619 07:25:34.034366 13157 impala-server.cc:286] Currently configured default
filesystem: ProxyLocalFileSystem. fs.defaultFS (file:/tmp) is not supported.
E0619 07:25:34.034384 13157 impala-server.cc:289] Aborting Impala Server
startup due to improper configuration. Impalad exiting.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7193) Local filesystem failes with fs.defaultFS (file:/tmp) is not supported
Tianyi Wang created IMPALA-7193:
---
Summary: Local filesystem failes with fs.defaultFS (file:/tmp) is
not supported
Key: IMPALA-7193
URL: https://issues.apache.org/jira/browse/IMPALA-7193
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Joe McDonnell
Impala failed to start on a local file system
{noformat}
I0619 07:25:34.034348 13157 status.cc:125] Currently configured default
filesystem: ProxyLocalFileSystem. fs.defaultFS (file:/tmp) is not supported.
@ 0x18a01f9 impala::Status::Status()
@ 0x1dd30b6 impala::Frontend::ValidateSettings()
@ 0x1dee2aa impala::ImpalaServer::ImpalaServer()
@ 0x1dea8db ImpaladMain()
@ 0x185c380 main
@ 0x7f373f829c05 __libc_start_main
@ 0x185c1f1 (unknown)
E0619 07:25:34.034366 13157 impala-server.cc:286] Currently configured default
filesystem: ProxyLocalFileSystem. fs.defaultFS (file:/tmp) is not supported.
E0619 07:25:34.034384 13157 impala-server.cc:289] Aborting Impala Server
startup due to improper configuration. Impalad exiting.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7192) test_timestamp_out_of_range doesn't fail on invalid timestamp
Tianyi Wang created IMPALA-7192:
---
Summary: test_timestamp_out_of_range doesn't fail on invalid
timestamp
Key: IMPALA-7192
URL: https://issues.apache.org/jira/browse/IMPALA-7192
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Taras Bobrovytsky
It's an ASAN build
{noformat}
07:29:23 TestParquet.test_timestamp_out_of_range[exec_option: {'batch_size':
0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
True, 'abort_on_error': 1, 'debug_action':
'-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
07:29:23 [gw6] linux2 -- Python 2.7.5
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/../infra/python/env/bin/python
07:29:23 query_test/test_scanners.py:316: in test_timestamp_out_of_range
07:29:23 vector, unique_database)
07:29:23 common/impala_test_suite.py:420: in run_test_case
07:29:23 assert False, "Expected exception: %s" % expected_str
07:29:23 E AssertionError: Expected exception: Parquet file
'hdfs://localhost:20500/test-warehouse/$DATABASE.db/out_of_range_timestamp/out_of_range_timestamp.parquet'
column 'ts' contains an out of range timestamp. The valid date range is
1400-01-01..-12-31.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514604#comment-16514604
]
Tianyi Wang commented on IMPALA-7169:
-
[~tarmstrong] To check the most recent checkpoint we need to parse the file
names in Trash and choose the one with the latest time stamp. HDFS might create
a new checkpoint/remove an old one concurrently so we still need to come up
with a way to lock the trash folder, or retry something in a loop, or else.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514596#comment-16514596
]
Tianyi Wang commented on IMPALA-7169:
-
[~tarmstrong] We can. But test_drop_partition_encrypt is not the only test that
looks into trash. The easier option it to set fs.trash.interval to a large
value - do you know any downside of doing so?
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/16/18 12:10 AM:
---
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day, which doesn't work. HDFS will do the checkpointing
at the next UNIX time which is a multiple of 1440 * 60 * 1000. The failed build
happened to run this test at around 17:00:00 so it failed.
was (Author: tianyiwang):
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day, which doesn't work. HDFS will do the checkpointing
at the next UNIX time which is a multiple of 1440 * 60. The failed build
happened to run this test at around 17:00:00 so it failed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/16/18 12:10 AM:
---
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day, which doesn't work. HDFS will do the checkpointing
at the next UNIX time which is a multiple of 1440 * 60. The failed build
happened to run this test at around 17:00:00 so it failed.
was (Author: tianyiwang):
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/15/18 11:53 PM:
---
We set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
was (Author: tianyiwang):
OK. So we set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang edited comment on IMPALA-7169 at 6/15/18 11:52 PM:
---
OK. So we set fs.trash.interval = 1440 so that HDFS supposedly shouldn't do the
checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
was (Author: tianyiwang):
OK. So cloudera's Hadoopa fs.trash.interval = 1440 so that HDFS supposedly
shouldn't do the checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514539#comment-16514539
]
Tianyi Wang commented on IMPALA-7169:
-
OK. So cloudera's Hadoopa fs.trash.interval = 1440 so that HDFS supposedly
shouldn't do the checkpointing within a day.
The build took less than a day but HDFS did it when this test is running.
My current theory is that there are some leftover in the trash in our HDFS
snapshot. The timestamp (which is part of the filename) on those leftover is
older than a day so that HDFS thought more than a day has passed.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514480#comment-16514480
]
Tianyi Wang commented on IMPALA-7169:
-
The log in the failed build:
{noformat}
2018-06-11 16:59:54,285 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile: /test-warehouse/test_enc
ryption_db.db/t1/j=3/j3.txt is closed by DFSClient_NONMAPREDUCE_-192125689_43
2018-06-11 17:00:00,067 INFO org.apache.hadoop.fs.TrashPolicyDefault:
TrashPolicyDefault#deleteCheckpoint fo
r trashRoot: hdfs://localhost:20500/user/jenkins/.Trash
2018-06-11 17:00:00,072 INFO org.apache.hadoop.fs.TrashPolicyDefault: Created
trash checkpoint: /user/jenkin
s/.Trash/18061117
2018-06-11 17:00:00,894 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:
jenkins (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/user/jenkins/.Trash/Current
/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt
{noformat}
The log of a success build
{noformat}
2018-06-13 22:20:24,736 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile: /test-warehouse/test_encryption_db.db/
t1/j=3/j3.txt is closed by DFSClient_NONMAPREDUCE_-2038519081_43
2018-06-13 22:20:31,100 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt
2018-06-13 22:20:31,108 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=1
2018-06-13 22:20:31,168 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073755231_14407 127.0.0.1:31000 127.0.0.1:310
01 127.0.0.1:31002
2018-06-13 22:20:33,664 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31001 to delete [blk_1073755231_14407]
2018-06-13 22:20:36,665 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31002 to delete [blk_1073755231_14407]
2018-06-13 22:20:37,098 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=2/j2.txt
2018-06-13 22:20:37,107 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=2
2018-06-13 22:20:37,167 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073755232_14408 127.0.0.1:31001 127.0.0.1:310
02 127.0.0.1:31000
2018-06-13 22:20:39,665 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31002 to delete [blk_1073755232_14408]
2018-06-13 22:20:42,666 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31001 to delete [blk_1073755232_14408]
2018-06-13 22:20:43,107 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=3/j3.txt
2018-06-13 22:20:43,117 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jenkins (auth:
SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/test-warehouse/test_encryption_db.db/t1/j=3
2018-06-13 22:20:45,666 INFO BlockStateChange: BLOCK* BlockManager: ask
127.0.0.1:31000 to delete [blk_1073755232_14408, b
lk_1073755231_14407]
2018-06-13 22:20:59,342 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073755230_14406 127.0.0.1:31001 127.0.0.1:310
02 127.0.0.1:31000
{noformat}
Maybe the "TrashPolicyDefault#deleteCheckpoint" operation raced with something.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
>
[jira] [Commented] (IMPALA-7154) Error making 'dropDatabase' RPC to Hive Metastore
[
https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514246#comment-16514246
]
Tianyi Wang commented on IMPALA-7154:
-
Saw it again. I took a brief look. The problem was that the connection to HMS
was lost after the drop database commend was sent. HMS client however didn't
know the command took effect and retried so it failed.
{noformat}
I0614 21:46:06.395942 14644 ClientCnxn.java:975] Opening socket connection to
server localhost/0:0:0:
0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
W0614 21:46:06.396255 14644 ClientCnxn.java:1102] Session 0x0 for server null,
unexpected error, clos
ing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
W0614 21:46:06.548285 12859 RetryingMetaStoreClient.java:148] MetaStoreClient
lost connection. Attemp
ting to reconnect.
Java exception follows:
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_drop_database(ThriftHiveMet
astore.java:733)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.drop_database(ThriftHiveMetastor
e.java:718)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:810)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101
)
at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
I0614 21:46:06.550910 12859 HiveMetaStoreClient.java:547] Closed a connection
to metastore, current c
onnections: 9
I0614 21:46:06.550966 12859 HiveMetaStoreClient.java:391] Trying to connect to
metastore with URI thr
ift://localhost:9083
I0614 21:46:06.551259 12859 HiveMetaStoreClient.java:465] Opened a connection
to metastore, current c
onnections: 10
I0614 21:46:06.551961 12859 HiveMetaStoreClient.java:518] Connected to
metastore.
I0614 21:46:06.555757 12859 jni-util.cc:230]
org.apache.impala.common.ImpalaRuntimeException: Error m
aking 'dropDatabase' RPC to Hive Metastore:
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: NoSuchObjectException(message:test_fuzz_nested_types_67367717)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16387)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16364)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMeta
store.java:16295)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMeta
store.java:702)
at
[jira] [Updated] (IMPALA-3040) test_caching_ddl failed with unexpected get_num_cache_requests
[
https://issues.apache.org/jira/browse/IMPALA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-3040:
Labels: broken-build flaky (was: flaky)
> test_caching_ddl failed with unexpected get_num_cache_requests
> --
>
> Key: IMPALA-3040
> URL: https://issues.apache.org/jira/browse/IMPALA-3040
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.5.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Major
> Labels: broken-build, flaky
>
> This test failed on the integration job.
> http://sandbox.jenkins.cloudera.com/view/Impala/view/Builds%20-%202.5.0%20release/job/impala-cdh5.7.0-integration/3/testReport/junit/query_test.test_hdfs_caching/TestHdfsCachingDdl/test_caching_ddl_exec_optiondisable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0table_format__text_none_/
> {code}
> query_test.test_hdfs_caching.TestHdfsCachingDdl.test_caching_ddl[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
> table_format: text/none] (from pytest)
> Failing for the past 1 build (Since Failed#3 )
> Took 47 sec.
> add description
> Error Message
> assert 9 == 10 + where 10 = get_num_cache_requests()
> Stacktrace
> query_test/test_hdfs_caching.py:132: in test_caching_ddl
> assert num_entries_pre == get_num_cache_requests()
> E assert 9 == 10
> E+ where 10 = get_num_cache_requests()
> Standard Error
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `cachedb` cascade;
> -- executing against localhost:21000
> create database cachedb;
> -- executing against localhost:21000
> use functional;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> use cachedb;
> -- executing against localhost:21000
> create table cached_tbl_nopart (i int) cached in 'testPool';
> -- executing against localhost:21000
> insert into cached_tbl_nopart select 1;
> -- executing against localhost:21000
> select * from cached_tbl_nopart;
> -- executing against localhost:21000
> create table like_cached_tbl_nopart like cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats like_cached_tbl_nopart;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> alter table cached_tbl_nopart set uncached;
> -- executing against localhost:21000
> show table stats cached_tbl_nopart;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool' with replication = 9;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> drop table if exists cached_tbl_part;
> -- executing against localhost:21000
> create table cached_tbl_part (i int) partitioned by (j int) cached in
> 'testPool';
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=0);
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=1) uncached;
> -- executing against localhost:21000
> alter table cached_tbl_part add partition (j=2) cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=2) set uncached;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=1) set cached in 'testPool';
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> -- executing against localhost:21000
> insert into cached_tbl_part partition(j) values(3, 3), (4, 4);
> -- executing against localhost:21000
> alter table cached_tbl_part partition (j=3) set cached in 'testPool' with
> replication = 4;
> -- executing against localhost:21000
> show partitions cached_tbl_part;
> --
[jira] [Resolved] (IMPALA-7177) Impalad hangs when starting up
[
https://issues.apache.org/jira/browse/IMPALA-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7177.
-
Resolution: Duplicate
> Impalad hangs when starting up
> --
>
> Key: IMPALA-7177
> URL: https://issues.apache.org/jira/browse/IMPALA-7177
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Lars Volker
>Priority: Critical
> Labels: broken-build
>
> In
> custom_cluster/test_breakpad.py::TestBreakpadExhaustive::test_minidump_cleanup_thread,
> a impalad hangs for 4 minutes when starting up. The tail of its log is:
> {noformat}
> I0613 14:55:55.514082 23644 init.cc:246] OS version: Linux version
> 2.6.32-358.14.1.el6.centos.plus.x86_64 (mockbu...@c6b9.bsys.dev.centos.org)
> (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Jul 16
> 21:33:24 UTC 2013
> Clock: clocksource: 'xen', clockid_t: CLOCK_MONOTONIC_COARSE
> I0613 14:55:55.514091 23644 init.cc:247] Process ID: 23644
> I0613 14:55:55.514098 23644 init.cc:248] Default AES cipher mode for
> spill-to-disk: AES-CTR
> I0613 14:55:58.153287 23644 llvm-codegen.cc:152] CPU class for runtime code
> generation: sandybridge
> I0613 14:55:58.153379 23644 llvm-codegen.cc:154] Detected CPU flags:
> +sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx512f,-clzero,-pku,+mmx,-lwp$
> I0613 14:55:58.153432 23644 llvm-codegen.cc:157] CPU flags enabled for
> runtime code generation:
> +sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx$
> I0613 14:56:00.633289 23644 GlogAppender.java:137] Logging (re)initialized.
> Impala: VLOG, All other: INFO
> I0613 14:56:00.638018 23644 JniFrontend.java:139] Authorization is 'DISABLED'.
> I0613 14:56:00.640646 23644 JniFrontend.java:141] Java Input arguments:
> -agentlib:jdwp=transport=dt_socket,address=30002,server=y,suspend=n
> -Djava.library.path=/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/Impala-Toolchain/cdh_components/hadoop-3.0.0-cdh6.x-417234//lib/native/
> -XX:ErrorFile=$
> Java System properties:
> awt.toolkit:sun.awt.X11.XToolkit
> file.encoding.pkg:sun.io
> java.specification.version:1.8
> sun.cpu.isalist:
> sun.jnu.encoding:UTF-8
> java.class.path:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/target/classes:/data/jenkins/workspace/impala-cdh6$
> I0613 14:56:00.691942 23644 HiveConf.java:188] Found configuration file
> file:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources/hive-site.xml
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-6642) start-impala-cluster.py failing in some custom cluster tests
[
https://issues.apache.org/jira/browse/IMPALA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang reopened IMPALA-6642:
-
Saw it again. This time in test_minidump_cleanup_thread.
> start-impala-cluster.py failing in some custom cluster tests
>
>
> Key: IMPALA-6642
> URL: https://issues.apache.org/jira/browse/IMPALA-6642
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.12.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Taras Bobrovytsky
>Priority: Critical
> Labels: broken-build
>
> Seen in recent builds, both on the 2.x branch. Not quite the same test/error,
> but they seem similar enough to probably be related:
> {noformat}
> 17:01:18 _ ERROR at setup of
> TestAdmissionController.test_require_user __
> 17:01:18 common/custom_cluster_test_suite.py:109: in setup_method
> 17:01:18 self._start_impala_cluster(cluster_args)
> 17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 17:01:18 check_call(cmd + options, close_fds=True)
> 17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 17:01:18 raise CalledProcessError(retcode, cmd)
> 17:01:18 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="-vmodule admission-controller=3
> -fair_scheduler_allocation_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml
> -llama_site_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml
> -disable_admission_control=false -require_username" ',
> '--state_store_args="-statestore_heartbeat_frequency_ms=100
> -statestore_priority_update_frequency_ms=100" ']' returned non-zero exit
> status 1
> 17:01:18 Captured stdout setup
> -
> 17:01:18 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 17:01:18 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found
> 17:01:18
> 17:01:18 Captured stderr setup
> -
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> {noformat}
> {noformat}
> 16:42:41 ___ ERROR at setup of
> TestAuthorization.test_access_runtime_profile
> 16:42:41 common/custom_cluster_test_suite.py:109: in setup_method
> 16:42:41 self._start_impala_cluster(cluster_args)
> 16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 16:42:41 check_call(cmd + options, close_fds=True)
> 16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 16:42:41 raise CalledProcessError(retcode, cmd)
> 16:42:41 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="--server_name=server1
> --authorization_policy_file=/test-warehouse/authz-policy.ini
> --authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status
> 1
> 16:42:41 Captured stdout setup
> -
> 16:42:41 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 16:42:41 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 16:42:41 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
>
[jira] [Created] (IMPALA-7177) Impalad hangs when starting up
Tianyi Wang created IMPALA-7177:
---
Summary: Impalad hangs when starting up
Key: IMPALA-7177
URL: https://issues.apache.org/jira/browse/IMPALA-7177
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Lars Volker
In
custom_cluster/test_breakpad.py::TestBreakpadExhaustive::test_minidump_cleanup_thread,
a impalad hangs for 4 minutes when starting up. The tail of its log is:
{noformat}
I0613 14:55:55.514082 23644 init.cc:246] OS version: Linux version
2.6.32-358.14.1.el6.centos.plus.x86_64 (mockbu...@c6b9.bsys.dev.centos.org)
(gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Jul 16
21:33:24 UTC 2013
Clock: clocksource: 'xen', clockid_t: CLOCK_MONOTONIC_COARSE
I0613 14:55:55.514091 23644 init.cc:247] Process ID: 23644
I0613 14:55:55.514098 23644 init.cc:248] Default AES cipher mode for
spill-to-disk: AES-CTR
I0613 14:55:58.153287 23644 llvm-codegen.cc:152] CPU class for runtime code
generation: sandybridge
I0613 14:55:58.153379 23644 llvm-codegen.cc:154] Detected CPU flags:
+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx512f,-clzero,-pku,+mmx,-lwp$
I0613 14:55:58.153432 23644 llvm-codegen.cc:157] CPU flags enabled for runtime
code generation:
+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx$
I0613 14:56:00.633289 23644 GlogAppender.java:137] Logging (re)initialized.
Impala: VLOG, All other: INFO
I0613 14:56:00.638018 23644 JniFrontend.java:139] Authorization is 'DISABLED'.
I0613 14:56:00.640646 23644 JniFrontend.java:141] Java Input arguments:
-agentlib:jdwp=transport=dt_socket,address=30002,server=y,suspend=n
-Djava.library.path=/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/Impala-Toolchain/cdh_components/hadoop-3.0.0-cdh6.x-417234//lib/native/
-XX:ErrorFile=$
Java System properties:
awt.toolkit:sun.awt.X11.XToolkit
file.encoding.pkg:sun.io
java.specification.version:1.8
sun.cpu.isalist:
sun.jnu.encoding:UTF-8
java.class.path:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/target/classes:/data/jenkins/workspace/impala-cdh6$
I0613 14:56:00.691942 23644 HiveConf.java:188] Found configuration file
file:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources/hive-site.xml
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7177) Impalad hangs when starting up
Tianyi Wang created IMPALA-7177:
---
Summary: Impalad hangs when starting up
Key: IMPALA-7177
URL: https://issues.apache.org/jira/browse/IMPALA-7177
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Lars Volker
In
custom_cluster/test_breakpad.py::TestBreakpadExhaustive::test_minidump_cleanup_thread,
a impalad hangs for 4 minutes when starting up. The tail of its log is:
{noformat}
I0613 14:55:55.514082 23644 init.cc:246] OS version: Linux version
2.6.32-358.14.1.el6.centos.plus.x86_64 (mockbu...@c6b9.bsys.dev.centos.org)
(gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Jul 16
21:33:24 UTC 2013
Clock: clocksource: 'xen', clockid_t: CLOCK_MONOTONIC_COARSE
I0613 14:55:55.514091 23644 init.cc:247] Process ID: 23644
I0613 14:55:55.514098 23644 init.cc:248] Default AES cipher mode for
spill-to-disk: AES-CTR
I0613 14:55:58.153287 23644 llvm-codegen.cc:152] CPU class for runtime code
generation: sandybridge
I0613 14:55:58.153379 23644 llvm-codegen.cc:154] Detected CPU flags:
+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx512f,-clzero,-pku,+mmx,-lwp$
I0613 14:55:58.153432 23644 llvm-codegen.cc:157] CPU flags enabled for runtime
code generation:
+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx$
I0613 14:56:00.633289 23644 GlogAppender.java:137] Logging (re)initialized.
Impala: VLOG, All other: INFO
I0613 14:56:00.638018 23644 JniFrontend.java:139] Authorization is 'DISABLED'.
I0613 14:56:00.640646 23644 JniFrontend.java:141] Java Input arguments:
-agentlib:jdwp=transport=dt_socket,address=30002,server=y,suspend=n
-Djava.library.path=/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/Impala-Toolchain/cdh_components/hadoop-3.0.0-cdh6.x-417234//lib/native/
-XX:ErrorFile=$
Java System properties:
awt.toolkit:sun.awt.X11.XToolkit
file.encoding.pkg:sun.io
java.specification.version:1.8
sun.cpu.isalist:
sun.jnu.encoding:UTF-8
java.class.path:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/target/classes:/data/jenkins/workspace/impala-cdh6$
I0613 14:56:00.691942 23644 HiveConf.java:188] Found configuration file
file:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources/hive-site.xml
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IMPALA-7175) In a local FS build, test_native_functions_race thinks there are 2 impalads where there should be 1
[
https://issues.apache.org/jira/browse/IMPALA-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-7175:
Summary: In a local FS build, test_native_functions_race thinks there are 2
impalads where there should be 1 (was: In a local build,
test_native_functions_race thinks there are 2 impalads where there should be 1)
> In a local FS build, test_native_functions_race thinks there are 2 impalads
> where there should be 1
> ---
>
> Key: IMPALA-7175
> URL: https://issues.apache.org/jira/browse/IMPALA-7175
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build
>
> In TestUdfExecution.test_native_functions_race, the test checks the number of
> impalads at the beginning and end of the test. In a local build there should
> be only 1 impalad but somehow the test found 2 at the beginning of the test
> and failed.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:379: in test_native_functions_race
> assert len(cluster.impalads) == exp_num_impalads
> E assert 1 == 2
> E+ where 1 = len([ 0xc9ffa90>])
> E+where [ 0xc9ffa90>] = 0x6a5d510>.impalads
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7175) In a local build, test_native_functions_race thinks there are 2 impalads where there should be 1
Tianyi Wang created IMPALA-7175:
---
Summary: In a local build, test_native_functions_race thinks there
are 2 impalads where there should be 1
Key: IMPALA-7175
URL: https://issues.apache.org/jira/browse/IMPALA-7175
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Vuk Ercegovac
In TestUdfExecution.test_native_functions_race, the test checks the number of
impalads at the beginning and end of the test. In a local build there should be
only 1 impalad but somehow the test found 2 at the beginning of the test and
failed.
{noformat}
Stacktrace
query_test/test_udfs.py:379: in test_native_functions_race
assert len(cluster.impalads) == exp_num_impalads
E assert 1 == 2
E+ where 1 = len([])
E+where [] = .impalads
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7175) In a local build, test_native_functions_race thinks there are 2 impalads where there should be 1
Tianyi Wang created IMPALA-7175:
---
Summary: In a local build, test_native_functions_race thinks there
are 2 impalads where there should be 1
Key: IMPALA-7175
URL: https://issues.apache.org/jira/browse/IMPALA-7175
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Vuk Ercegovac
In TestUdfExecution.test_native_functions_race, the test checks the number of
impalads at the beginning and end of the test. In a local build there should be
only 1 impalad but somehow the test found 2 at the beginning of the test and
failed.
{noformat}
Stacktrace
query_test/test_udfs.py:379: in test_native_functions_race
assert len(cluster.impalads) == exp_num_impalads
E assert 1 == 2
E+ where 1 = len([])
E+where [] = .impalads
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
Tianyi Wang created IMPALA-7174:
---
Summary: TestAdmissionController.test_cancellation failed with
incorrect total-admitted metric
Key: IMPALA-7174
URL: https://issues.apache.org/jira/browse/IMPALA-7174
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tim Armstrong
The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an descendant
of 'IMPALA-5216: Make admission control queuing async'.
{noformat}
Stacktrace
custom_cluster/test_admission_controller.py:557: in test_cancellation
assert self.cluster.impalads[0].service.get_metric_value(
E assert 0 == 3
E+ where 0 = >('admission-controller.total-admitted.default-pool')
E+where > =
.get_metric_value
E+ where = .service
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
Tianyi Wang created IMPALA-7174:
---
Summary: TestAdmissionController.test_cancellation failed with
incorrect total-admitted metric
Key: IMPALA-7174
URL: https://issues.apache.org/jira/browse/IMPALA-7174
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tim Armstrong
The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an descendant
of 'IMPALA-5216: Make admission control queuing async'.
{noformat}
Stacktrace
custom_cluster/test_admission_controller.py:557: in test_cancellation
assert self.cluster.impalads[0].service.get_metric_value(
E assert 0 == 3
E+ where 0 = >('admission-controller.total-admitted.default-pool')
E+where > =
.get_metric_value
E+ where = .service
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-7169) TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
[
https://issues.apache.org/jira/browse/IMPALA-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511888#comment-16511888
]
Tianyi Wang commented on IMPALA-7169:
-
1. This tests claims that Hadoop 2.8+ exhibit certain behavior but when I was
experimenting it's the same for Hadoop 2.6 as well. But this is not the reason
the build failed.
2. The problem is Impala removed a partition but it didn't appear in trash. I
don't know how consistent namenode is but I suspect something is asynchronous
with moving a encrypted file. I will look into HDFS code.
> TestHdfsEncryption::()::test_drop_partition_encrypt fails to find file
> --
>
> Key: IMPALA-7169
> URL: https://issues.apache.org/jira/browse/IMPALA-7169
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Lars Volker
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> F
> metadata/test_hdfs_encryption.py::TestHdfsEncryption::()::test_drop_partition_encrypt
> metadata/test_hdfs_encryption.py:202: in test_drop_partition_encrypt
> assert self.hdfs_client.exists(
> E assert 0xba6ee90>>('/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt')
> E+ where > =
> .exists
> E+where 0xba6ee90> = 0xba6ed50>.hdfs_client
> E+ and
> '/user/jenkins/.Trash/Current/test-warehouse/test_encryption_db.db/t1/j=1/j1.txt'
> = ('jenkins',
> 'test_encryption_db')
> E+where =
> '/user/{0}/.Trash/Current/test-warehouse/{1}.db/t1/j=1/j1.txt'.format
> E+and 'jenkins' = ()
> E+ where = getpass.getuser
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7170) test/comparison is broken
[
https://issues.apache.org/jira/browse/IMPALA-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-7170:
Component/s: Infrastructure
> test/comparison is broken
> -
>
> Key: IMPALA-7170
> URL: https://issues.apache.org/jira/browse/IMPALA-7170
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0
>Reporter: Tianyi Wang
>Priority: Major
>
> test/comparison in Impala 3.x is broken, presumably by the switch to Hadoop 3.
> Firstly, to run the tests in impala 3.x, the mini-cluster needs to be started
> with YARN, which is not documented anywhere.
> Then, data_generator.py will exit with the following error:
> {noformat}
> 2018-04-23 23:15:46,065 INFO:db_connection[752]:Dropping database randomness
> 2018-04-23 23:15:46,095 INFO:db_connection[234]:Creating database randomness
> 2018-04-23 23:15:52,390 INFO:data_generator[235]:Starting MR job to generate
> data for randomness
> Traceback (most recent call last):
> File "tests/comparison/data_generator.py", line 339, in
> populator.populate_db(args.table_count, postgresql_conn=postgresql_conn)
> File "tests/comparison/data_generator.py", line 134, in populate_db
> self._run_data_generator_mr_job([g for _, g in table_and_generators],
> self.db_name)
> File "tests/comparison/data_generator.py", line 244, in
> _run_data_generator_mr_job
> % (reducer_count, ','.join(files), mapper_input_file, hdfs_output_dir))
> File "/home/impdev/projects/impala/tests/comparison/cluster.py", line 476,
> in run_mr_job
> stderr=subprocess.STDOUT, env=env)
> File "/home/impdev/projects/impala/tests/util/shell_util.py", line 113, in
> shell
> "\ncmd: %s\nstdout: %s\nstderr: %s") % (retcode, cmd, output, err))
> Exception: Command returned non-zero exit code: 1
> cmd: set -euo pipefail
> hadoop jar
> /home/impdev/projects/impala/toolchain/cdh_components/hadoop-3.0.0-cdh6.x-SNAPSHOT/share/hadoop/tools/lib/hadoop-streaming-3.0.0-cdh6.x-SNAPSHOT.jar
> -D mapred.reduce.tasks=36 \
> -D stream.num.map.output.key.fields=2 \
> -files
> tests/comparison/common.py,tests/comparison/db_types.py,tests/comparison/data_generator_mapred_common.py,tests/comparison/data_generator_mapper.py,tests/comparison/data_generator_reducer.py,tests/comparison/random_val_generator.py
> \
> -input /tmp/data_gen_randomness_mr_input_1524525348 \
> -output /tmp/data_gen_randomness_mr_output_1524525348 \
> -mapper data_generator_mapper.py \
> -reducer data_generator_reducer.py
> stdout: packageJobJar: []
> [/home/impdev/projects/impala/toolchain/cdh_components/hadoop-3.0.0-cdh6.x-SNAPSHOT/share/hadoop/tools/lib/hadoop-streaming-3.0.0-cdh6.x-SNAPSHOT.jar]
> /tmp/streamjob2990195923122538287.jar tmpDir=null
> 18/04/23 23:15:53 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 18/04/23 23:15:53 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 18/04/23 23:15:54 INFO mapreduce.JobResourceUploader: Disabling Erasure
> Coding for path:
> /tmp/hadoop-yarn/staging/impdev/.staging/job_1524519161700_0002
> 18/04/23 23:15:54 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 18/04/23 23:15:54 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library [hadoop-lzo rev 2b3bd7731ff3ef5d8585a004b90696630e5cea96]
> 18/04/23 23:15:54 INFO mapred.FileInputFormat: Total input files to process :
> 1
> 18/04/23 23:15:54 INFO mapreduce.JobSubmitter: number of splits:2
> 18/04/23 23:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 18/04/23 23:15:54 INFO Configuration.deprecation:
> yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead,
> use yarn.system-metrics-publisher.enabled
> 18/04/23 23:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1524519161700_0002
> 18/04/23 23:15:54 INFO mapreduce.JobSubmitter: Executing with tokens: []
> 18/04/23 23:15:54 INFO conf.Configuration: resource-types.xml not found
> 18/04/23 23:15:54 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 18/04/23 23:15:54 INFO impl.YarnClientImpl: Submitted application
> application_1524519161700_0002
> 18/04/23 23:15:54 INFO mapreduce.Job: The url to track the job:
> http://c37e0835e988:8088/proxy/application_1524519161700_0002/
> 18/04/23 23:15:54 INFO mapreduce.Job: Running job: job_1524519161700_0002
> 18/04/23 23:16:00 INFO mapreduce.Job: Job job_1524519161700_0002 running in
> uber mode : false
> 18/04/23 23:16:00 INFO mapreduce.Job: map 0% reduce 0%
> 18/04/23 23:16:06 INFO mapreduce.Job: Job job_1524519161700_0002 failed with
> state FAILED due to: Application application_1524519161700_0002 failed 2
> times due to AM Container
[jira] [Created] (IMPALA-7170) test/comparison is broken
Tianyi Wang created IMPALA-7170:
---
Summary: test/comparison is broken
Key: IMPALA-7170
URL: https://issues.apache.org/jira/browse/IMPALA-7170
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 3.0
Reporter: Tianyi Wang
test/comparison in Impala 3.x is broken, presumably by the switch to Hadoop 3.
Firstly, to run the tests in impala 3.x, the mini-cluster needs to be started
with YARN, which is not documented anywhere.
Then, data_generator.py will exit with the following error:
{noformat}
2018-04-23 23:15:46,065 INFO:db_connection[752]:Dropping database randomness
2018-04-23 23:15:46,095 INFO:db_connection[234]:Creating database randomness
2018-04-23 23:15:52,390 INFO:data_generator[235]:Starting MR job to generate
data for randomness
Traceback (most recent call last):
File "tests/comparison/data_generator.py", line 339, in
populator.populate_db(args.table_count, postgresql_conn=postgresql_conn)
File "tests/comparison/data_generator.py", line 134, in populate_db
self._run_data_generator_mr_job([g for _, g in table_and_generators],
self.db_name)
File "tests/comparison/data_generator.py", line 244, in
_run_data_generator_mr_job
% (reducer_count, ','.join(files), mapper_input_file, hdfs_output_dir))
File "/home/impdev/projects/impala/tests/comparison/cluster.py", line 476, in
run_mr_job
stderr=subprocess.STDOUT, env=env)
File "/home/impdev/projects/impala/tests/util/shell_util.py", line 113, in
shell
"\ncmd: %s\nstdout: %s\nstderr: %s") % (retcode, cmd, output, err))
Exception: Command returned non-zero exit code: 1
cmd: set -euo pipefail
hadoop jar
/home/impdev/projects/impala/toolchain/cdh_components/hadoop-3.0.0-cdh6.x-SNAPSHOT/share/hadoop/tools/lib/hadoop-streaming-3.0.0-cdh6.x-SNAPSHOT.jar
-D mapred.reduce.tasks=36 \
-D stream.num.map.output.key.fields=2 \
-files
tests/comparison/common.py,tests/comparison/db_types.py,tests/comparison/data_generator_mapred_common.py,tests/comparison/data_generator_mapper.py,tests/comparison/data_generator_reducer.py,tests/comparison/random_val_generator.py
\
-input /tmp/data_gen_randomness_mr_input_1524525348 \
-output /tmp/data_gen_randomness_mr_output_1524525348 \
-mapper data_generator_mapper.py \
-reducer data_generator_reducer.py
stdout: packageJobJar: []
[/home/impdev/projects/impala/toolchain/cdh_components/hadoop-3.0.0-cdh6.x-SNAPSHOT/share/hadoop/tools/lib/hadoop-streaming-3.0.0-cdh6.x-SNAPSHOT.jar]
/tmp/streamjob2990195923122538287.jar tmpDir=null
18/04/23 23:15:53 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
18/04/23 23:15:53 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
18/04/23 23:15:54 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /tmp/hadoop-yarn/staging/impdev/.staging/job_1524519161700_0002
18/04/23 23:15:54 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
18/04/23 23:15:54 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 2b3bd7731ff3ef5d8585a004b90696630e5cea96]
18/04/23 23:15:54 INFO mapred.FileInputFormat: Total input files to process : 1
18/04/23 23:15:54 INFO mapreduce.JobSubmitter: number of splits:2
18/04/23 23:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
18/04/23 23:15:54 INFO Configuration.deprecation:
yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead,
use yarn.system-metrics-publisher.enabled
18/04/23 23:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1524519161700_0002
18/04/23 23:15:54 INFO mapreduce.JobSubmitter: Executing with tokens: []
18/04/23 23:15:54 INFO conf.Configuration: resource-types.xml not found
18/04/23 23:15:54 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
18/04/23 23:15:54 INFO impl.YarnClientImpl: Submitted application
application_1524519161700_0002
18/04/23 23:15:54 INFO mapreduce.Job: The url to track the job:
http://c37e0835e988:8088/proxy/application_1524519161700_0002/
18/04/23 23:15:54 INFO mapreduce.Job: Running job: job_1524519161700_0002
18/04/23 23:16:00 INFO mapreduce.Job: Job job_1524519161700_0002 running in
uber mode : false
18/04/23 23:16:00 INFO mapreduce.Job: map 0% reduce 0%
18/04/23 23:16:06 INFO mapreduce.Job: Job job_1524519161700_0002 failed with
state FAILED due to: Application application_1524519161700_0002 failed 2 times
due to AM Container for appattempt_1524519161700_0002_02 exited with
exitCode: 255
Failing this attempt.Diagnostics: [2018-04-23 23:16:06.473]Exception from
container-launch.
Container id: container_1524519161700_0002_02_01
Exit code: 255
[2018-04-23 23:16:06.475]Container exited with a non-zero exit code 255. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
[jira] [Resolved] (IMPALA-7167) Catalogd should reject files with unsupported formats
[
https://issues.apache.org/jira/browse/IMPALA-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7167.
-
Resolution: Not A Problem
[~tarmstrong] Thanks! I will close this.
> Catalogd should reject files with unsupported formats
> -
>
> Key: IMPALA-7167
> URL: https://issues.apache.org/jira/browse/IMPALA-7167
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Tianyi Wang
>Priority: Minor
>
> Currently when a file format is unsupported, impalad will log an error:
> {noformat}
> E0611 13:42:47.428393 22076 ImpaladCatalog.java:201] Error adding catalog
> object: Expected compressed text file with {.lzo,.gzip,.snappy,.bz2} suffix:
> 00_0.deflate
> Java exception follows:
> java.lang.RuntimeException: Expected compressed text file with
> {.lzo,.gzip,.snappy,.bz2} suffix: 00_0.deflate
> at
> org.apache.impala.catalog.HdfsPartition.(HdfsPartition.java:772)
> at
> org.apache.impala.catalog.HdfsPartition.fromThrift(HdfsPartition.java:884)
> at
> org.apache.impala.catalog.HdfsTable.loadFromThrift(HdfsTable.java:1678)
> at org.apache.impala.catalog.Table.fromThrift(Table.java:311)
> at
> org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:403)
> at
> org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:292)
> at
> org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:199)
> at
> org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:228)
> at
> org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:174)
> {noformat}
> Catalogd should filter out unsupported files instead of letting every impalad
> log the error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7167) Catalogd should reject files with unsupported formats
Tianyi Wang created IMPALA-7167:
---
Summary: Catalogd should reject files with unsupported formats
Key: IMPALA-7167
URL: https://issues.apache.org/jira/browse/IMPALA-7167
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: Tianyi Wang
Currently when a file format is unsupported, impalad will log an error:
{noformat}
E0611 13:42:47.428393 22076 ImpaladCatalog.java:201] Error adding catalog
object: Expected compressed text file with {.lzo,.gzip,.snappy,.bz2} suffix:
00_0.deflate
Java exception follows:
java.lang.RuntimeException: Expected compressed text file with
{.lzo,.gzip,.snappy,.bz2} suffix: 00_0.deflate
at
org.apache.impala.catalog.HdfsPartition.(HdfsPartition.java:772)
at
org.apache.impala.catalog.HdfsPartition.fromThrift(HdfsPartition.java:884)
at
org.apache.impala.catalog.HdfsTable.loadFromThrift(HdfsTable.java:1678)
at org.apache.impala.catalog.Table.fromThrift(Table.java:311)
at
org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:403)
at
org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:292)
at
org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:199)
at
org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:228)
at
org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:174)
{noformat}
Catalogd should filter out unsupported files instead of letting every impalad
log the error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7167) Catalogd should reject files with unsupported formats
Tianyi Wang created IMPALA-7167:
---
Summary: Catalogd should reject files with unsupported formats
Key: IMPALA-7167
URL: https://issues.apache.org/jira/browse/IMPALA-7167
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: Tianyi Wang
Currently when a file format is unsupported, impalad will log an error:
{noformat}
E0611 13:42:47.428393 22076 ImpaladCatalog.java:201] Error adding catalog
object: Expected compressed text file with {.lzo,.gzip,.snappy,.bz2} suffix:
00_0.deflate
Java exception follows:
java.lang.RuntimeException: Expected compressed text file with
{.lzo,.gzip,.snappy,.bz2} suffix: 00_0.deflate
at
org.apache.impala.catalog.HdfsPartition.(HdfsPartition.java:772)
at
org.apache.impala.catalog.HdfsPartition.fromThrift(HdfsPartition.java:884)
at
org.apache.impala.catalog.HdfsTable.loadFromThrift(HdfsTable.java:1678)
at org.apache.impala.catalog.Table.fromThrift(Table.java:311)
at
org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:403)
at
org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:292)
at
org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:199)
at
org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:228)
at
org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:174)
{noformat}
Catalogd should filter out unsupported files instead of letting every impalad
log the error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IMPALA-3816) Codegen perf-critical loops in Sorter
[
https://issues.apache.org/jira/browse/IMPALA-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-3816:
Issue Type: Improvement (was: Bug)
> Codegen perf-critical loops in Sorter
> -
>
> Key: IMPALA-3816
> URL: https://issues.apache.org/jira/browse/IMPALA-3816
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.7.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Minor
> Labels: codegen
> Attachments: percentile query profile.txt, tpch_30.txt
>
>
> In the sorter, we codegen the comparator function but call it indirectly via
> a function pointer. We should consider codegening the perf-critical loops so
> that we can make the comparator function call direct and inlinable. Inlining
> the comparison will be very beneficial if it is trivial, e.g. order by a
> numeric column: I expect sorts on simple keys will get noticably faster.
> We should also be able to get rid of FreeLocalAllocations() calls for most
> comparators, although I'm not sure what the best way to approach that is.
> The Partition() loop is the most perf-critical, followed by InsertionSort().
> We also don't do this yet for the TopN node, see IMPALA-3815.
> Mostafa's analysis:
> While evaluating Sort performance I noticed that the codegened compare
> function is not inlined which results in large overhead per row.
> Expected speedup is 10-15%
> {code}
> /// Returns a negative value if lhs is less than rhs, a positive value if
> lhs is
> /// greater than rhs, or 0 if they are equal. All exprs
> (ordering_exprs_lhs_ and
> /// ordering_exprs_rhs_) must have been prepared and opened before calling
> this,
> /// i.e. 'sort_key_exprs' in the constructor must have been opened.
> int ALWAYS_INLINE Compare(const TupleRow* lhs, const TupleRow* rhs) const {
> return codegend_compare_fn_ == NULL ?
> CompareInterpreted(lhs, rhs) :
> (*codegend_compare_fn_)(ordering_expr_evals_lhs_.data(),
> ordering_expr_evals_rhs_.data(), lhs, rhs);
> }
> {code}
> From Perf
> {code}
> │bool Sorter::TupleSorter::Less(const TupleRow* lhs, const
> TupleRow* rhs) {
>
>▒
> 7.43 │ push %rbp
>
>
>▒
> 3.23 │ mov%rsp,%rbp
>
>
>▒
> 9.44 │ push %r12
>
>
>▒
> 2.69 │ push %rbx
>
>
>▒
> 3.89 │ mov%rsi,%r12
>
>
>▒
> 2.98 │ mov%rdi,%rbx
>
>
>▒
> 6.06 │ sub$0x10,%rsp
>
>
>◆
>│ --num_comparisons_till_free_;
>
>
>▒
>│ DCHECK_GE(num_comparisons_till_free_, 0);
>
>
>▒
>│ if (UNLIKELY(num_comparisons_till_free_ == 0)) {
>
[jira] [Updated] (IMPALA-3816) Codegen perf-critical loops in Sorter
[
https://issues.apache.org/jira/browse/IMPALA-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-3816:
Attachment: tpch_30.txt
> Codegen perf-critical loops in Sorter
> -
>
> Key: IMPALA-3816
> URL: https://issues.apache.org/jira/browse/IMPALA-3816
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.7.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Minor
> Labels: codegen
> Attachments: percentile query profile.txt, tpch_30.txt
>
>
> In the sorter, we codegen the comparator function but call it indirectly via
> a function pointer. We should consider codegening the perf-critical loops so
> that we can make the comparator function call direct and inlinable. Inlining
> the comparison will be very beneficial if it is trivial, e.g. order by a
> numeric column: I expect sorts on simple keys will get noticably faster.
> We should also be able to get rid of FreeLocalAllocations() calls for most
> comparators, although I'm not sure what the best way to approach that is.
> The Partition() loop is the most perf-critical, followed by InsertionSort().
> We also don't do this yet for the TopN node, see IMPALA-3815.
> Mostafa's analysis:
> While evaluating Sort performance I noticed that the codegened compare
> function is not inlined which results in large overhead per row.
> Expected speedup is 10-15%
> {code}
> /// Returns a negative value if lhs is less than rhs, a positive value if
> lhs is
> /// greater than rhs, or 0 if they are equal. All exprs
> (ordering_exprs_lhs_ and
> /// ordering_exprs_rhs_) must have been prepared and opened before calling
> this,
> /// i.e. 'sort_key_exprs' in the constructor must have been opened.
> int ALWAYS_INLINE Compare(const TupleRow* lhs, const TupleRow* rhs) const {
> return codegend_compare_fn_ == NULL ?
> CompareInterpreted(lhs, rhs) :
> (*codegend_compare_fn_)(ordering_expr_evals_lhs_.data(),
> ordering_expr_evals_rhs_.data(), lhs, rhs);
> }
> {code}
> From Perf
> {code}
> │bool Sorter::TupleSorter::Less(const TupleRow* lhs, const
> TupleRow* rhs) {
>
>▒
> 7.43 │ push %rbp
>
>
>▒
> 3.23 │ mov%rsp,%rbp
>
>
>▒
> 9.44 │ push %r12
>
>
>▒
> 2.69 │ push %rbx
>
>
>▒
> 3.89 │ mov%rsi,%r12
>
>
>▒
> 2.98 │ mov%rdi,%rbx
>
>
>▒
> 6.06 │ sub$0x10,%rsp
>
>
>◆
>│ --num_comparisons_till_free_;
>
>
>▒
>│ DCHECK_GE(num_comparisons_till_free_, 0);
>
>
>▒
>│ if (UNLIKELY(num_comparisons_till_free_ == 0)) {
>
[jira] [Assigned] (IMPALA-7097) Print EC info in the query plan and profile
[
https://issues.apache.org/jira/browse/IMPALA-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang reassigned IMPALA-7097:
---
Assignee: (was: Tianyi Wang)
> Print EC info in the query plan and profile
> ---
>
> Key: IMPALA-7097
> URL: https://issues.apache.org/jira/browse/IMPALA-7097
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Priority: Minor
>
> Impala should print EC-related info in the query plan to help user understand
> the behavior and diagnose performance issues easier. The most trivial design
> would look like:
> {noformat}
> [localhost:21000] functional> explain select * from functional.alltypes;
> Query: explain select * from functional.alltypes
> +-+
> | Explain String |
> +-+
> | Max Per-Host Resource Reservation: Memory=32.00KB Threads=3 |
> | Per-Host Resource Estimates: Memory=160.00MB|
> | Codegen disabled by planner |
> | |
> | PLAN-ROOT SINK |
> | | |
> | 01:EXCHANGE [UNPARTITIONED] |
> | | |
> | 00:SCAN HDFS [functional.alltypes] |
> |partitions=24/24 files=25 size=498.41KB |
> |EC files=24 size=478.45KB|
> +-+
> {noformat}
> In the query profile we should at least print "EC bytes scanned" in the scan
> node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7092) Re-enable EC tests broken by HDFS-13539
[ https://issues.apache.org/jira/browse/IMPALA-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang reassigned IMPALA-7092: --- Assignee: (was: Tianyi Wang) > Re-enable EC tests broken by HDFS-13539 > > > Key: IMPALA-7092 > URL: https://issues.apache.org/jira/browse/IMPALA-7092 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend, Infrastructure >Affects Versions: Impala 3.1.0 >Reporter: Tianyi Wang >Priority: Major > > With HDFS-13539 and HDFS-13540 fixed, we should be able to re-enable some > tests and diagnose the causes of the remaining failed tests without much > noise. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Issue Comment Deleted] (IMPALA-4065) Inline comparator calls into TopN::InsertBatch()
[ https://issues.apache.org/jira/browse/IMPALA-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyi Wang updated IMPALA-4065: Comment: was deleted (was: I observed more than 2X speed up in the TopNNode with an "order by int_col" extreme case, but not so much for the sort case in IMPALA-3816. It seems LLVM doesn't optimize the template code well if there are indirect calls among them. ) > Inline comparator calls into TopN::InsertBatch() > > > Key: IMPALA-4065 > URL: https://issues.apache.org/jira/browse/IMPALA-4065 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Assignee: Tianyi Wang >Priority: Minor > Labels: codegen, ramp-up > > This is the more interesting follow-on from IMPALA-3815. We should inline the > Compare() calls in the codegen'd TopN code to avoid the indirect function > pointer call. > The tricky aspect is that the Compare() calls are called from > std::priority_queue, and we don't have a way to force-inline those functions > at the moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4065) Inline comparator calls into TopN::InsertBatch()
[ https://issues.apache.org/jira/browse/IMPALA-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506691#comment-16506691 ] Tianyi Wang commented on IMPALA-4065: - I observed more than 2X speed up in the TopNNode with an "order by int_col" extreme case, but not so much for the sort case in IMPALA-3816. It seems LLVM doesn't optimize the template code well if there are indirect calls among them. > Inline comparator calls into TopN::InsertBatch() > > > Key: IMPALA-4065 > URL: https://issues.apache.org/jira/browse/IMPALA-4065 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Assignee: Tianyi Wang >Priority: Minor > Labels: codegen, ramp-up > > This is the more interesting follow-on from IMPALA-3815. We should inline the > Compare() calls in the codegen'd TopN code to avoid the indirect function > pointer call. > The tricky aspect is that the Compare() calls are called from > std::priority_queue, and we don't have a way to force-inline those functions > at the moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3926) Reconsider use of LD_LIBRARY_PATH for toolchain libraries
[
https://issues.apache.org/jira/browse/IMPALA-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506637#comment-16506637
]
Tianyi Wang commented on IMPALA-3926:
-
For FE tests we can simply use -Djava.library.path. I think we only need
libfesupport.so for FE tests.
We do have dynamic dependencies not located at FHS locations. For example:
{noformat}
export LD_LIBRARY_PATH=
ldd protobuf-2.6.1/bin/protoc
linux-vdso.so.1 => (0x7fff70f9f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x7f5af9a11000)
libprotobuf.so.9 => not found
libprotoc.so.9 => not found
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f5af97f7000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f5af94ee000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x7f5af92d8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f5af8f0e000)
/lib64/ld-linux-x86-64.so.2 (0x7f5af9c2e000)
{noformat}
They can be collected using ldd in a clean shell:
{noformat}
export LD_LIBRARY_PATH=
find -type f -executable | xargs ldd | grep "not found"
find -name "*.so" | xargs ldd | grep "not found"
{noformat}
For those dependencies one option is to patch the rpath after we download the
toolchain using tools like https://nixos.org/patchelf.html
Another option is to install toolchains into a well known location. I think
these are easier to reason about and less likely to backfire than setting
LD_LIBRARY_PATH for individual binaries.
> Reconsider use of LD_LIBRARY_PATH for toolchain libraries
> -
>
> Key: IMPALA-3926
> URL: https://issues.apache.org/jira/browse/IMPALA-3926
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 2.6.0
>Reporter: Matthew Jacobs
>Priority: Major
> Labels: build, toolchain
>
> Right now, impala-config.sh puts a lot of libraries in LD_LIBRARY_PATH, but
> this can be a problem for binaries that aren't from our builds or explicitly
> built against these specific libraries. One solution is to move any tools we
> need into the toolchain and build against these libraries. While this may be
> a reasonable thing to do (i.e. moving all tools we need into the toolchain),
> we should consider if setting LD_LIBRARY_PATH for the whole Impala
> environment is really necessary and the right thing to do (e.g. [some people
> say using LD_LIBRARY_PATH is
> bad|http://xahlee.info/UnixResource_dir/_/ldpath.html]).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-5990) End-to-end compression of metadata
[ https://issues.apache.org/jira/browse/IMPALA-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506446#comment-16506446 ] Tianyi Wang edited comment on IMPALA-5990 at 6/8/18 6:51 PM: - Today I learned that a thrift message larger than 4GB can be used with TBufferedTransport and TBinaryProtocol at least in C++. The limits are at other places: TMemoryBuffer cannot handle a message larger than 4GB, thrift cannot handle a single std::string larger than 4GB, etc. So after IMPALA-5990, we have seen ~6GB compressed catalog and it works just fine. was (Author: tianyiwang): Today I learned that a thrift message larger than 4GB can be used with TBufferedTransport and TBinaryProtocol. The limits are at other places: TMemoryBuffer cannot handle a message larger than 4GB, thrift cannot handle a single std::string larger than 4GB, etc. So after IMPALA-5990, we have seen ~6GB compressed catalog and it works just fine. > End-to-end compression of metadata > -- > > Key: IMPALA-5990 > URL: https://issues.apache.org/jira/browse/IMPALA-5990 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Frontend >Affects Versions: Impala 2.8.0, Impala 2.9.0, Impala 2.10.0 >Reporter: Alexander Behm >Assignee: Tianyi Wang >Priority: Critical > Fix For: Impala 2.12.0 > > > The metadata of large tables can become quite big making it costly to hold in > the statestore and disseminate to coordinator impalads. The metadata can even > get so big that fundamental limits like the JVM 2GB array size and the Thrift > 4GB are hit and lead to downtime. > For reducing the statestore metadata topic size we have an existing > "compact_catalog_topic" flag which LZ4 compresses the metadata payload for > the C++ codepaths catalogd->statestore and statestore->impalad. > Unfortunately, the metadata is not compressed in the same way during the > FE->BE transition on the catalogd and the BE->FE transition on the impalad. > The goal of this change is to enable end-to-end compression for the full path > of metadata dissemination. The existing code paths also need significant > cleanup/streamlining. Ideally, the new code should provide consistent size > limits everywhere. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5990) End-to-end compression of metadata
[ https://issues.apache.org/jira/browse/IMPALA-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506446#comment-16506446 ] Tianyi Wang commented on IMPALA-5990: - Today I learned that a thrift message larger than 4GB can be used with TBufferedTransport and TBinaryProtocol. The limits are at other places: TMemoryBuffer cannot handle a message larger than 4GB, thrift cannot handle a single std::string larger than 4GB, etc. So after IMPALA-5990, we have seen ~6GB compressed catalog and it works just fine. > End-to-end compression of metadata > -- > > Key: IMPALA-5990 > URL: https://issues.apache.org/jira/browse/IMPALA-5990 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Frontend >Affects Versions: Impala 2.8.0, Impala 2.9.0, Impala 2.10.0 >Reporter: Alexander Behm >Assignee: Tianyi Wang >Priority: Critical > Fix For: Impala 2.12.0 > > > The metadata of large tables can become quite big making it costly to hold in > the statestore and disseminate to coordinator impalads. The metadata can even > get so big that fundamental limits like the JVM 2GB array size and the Thrift > 4GB are hit and lead to downtime. > For reducing the statestore metadata topic size we have an existing > "compact_catalog_topic" flag which LZ4 compresses the metadata payload for > the C++ codepaths catalogd->statestore and statestore->impalad. > Unfortunately, the metadata is not compressed in the same way during the > FE->BE transition on the catalogd and the BE->FE transition on the impalad. > The goal of this change is to enable end-to-end compression for the full path > of metadata dissemination. The existing code paths also need significant > cleanup/streamlining. Ideally, the new code should provide consistent size > limits everywhere. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4065) Inline comparator calls into TopN::InsertBatch()
[ https://issues.apache.org/jira/browse/IMPALA-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505448#comment-16505448 ] Tianyi Wang commented on IMPALA-4065: - [~zoram] If you are not working on this I can take it over. I'm working on IMPALA-3816 and will refactor some code around this. > Inline comparator calls into TopN::InsertBatch() > > > Key: IMPALA-4065 > URL: https://issues.apache.org/jira/browse/IMPALA-4065 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Assignee: Zoram Thanga >Priority: Minor > Labels: codegen, ramp-up > > This is the more interesting follow-on from IMPALA-3815. We should inline the > Compare() calls in the codegen'd TopN code to avoid the indirect function > pointer call. > The tricky aspect is that the Compare() calls are called from > std::priority_queue, and we don't have a way to force-inline those functions > at the moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-3040) test_caching_ddl failed with unexpected get_num_cache_requests
[
https://issues.apache.org/jira/browse/IMPALA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504136#comment-16504136
]
Tianyi Wang edited comment on IMPALA-3040 at 6/7/18 8:43 PM:
-
In the case I looked into the problem is that the cache directive
/test-warehouse/cachedb.db/cached_tbl_local/j=1 never got removed. The relevant
namenode log:
{noformat}
2018-05-29 22:31:55,636 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3000 milliseconds
2018-05-29 22:31:55,637 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
14 directive(s) and 7 block(s) in 1 millisecond(s).
2018-05-29 22:31:58,636 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3000 milliseconds
2018-05-29 22:31:58,637 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
14 directive(s) and 7 block(s) in 1 millisecond(s).
2018-05-29 22:32:01,636 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3000 milliseconds
2018-05-29 22:32:01,637 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
14 directive(s) and 7 block(s) in 1 millisecond(s).
2018-05-29 22:32:03,483 INFO
org.apache.hadoop.hdfs.server.namenode.CacheManager: addDirective of {path:
/test-warehouse/cachedb.db/cached_tbl_local/j=1, replication: 1, pool:
testPool, expiration: 26687997791:19:48:13.951} successful.
2018-05-29 22:32:03,679 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning because of pending operations
2018-05-29 22:32:03,680 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
15 directive(s) and 7 block(s) in 1 millisecond(s).
2018-05-29 22:32:04,738 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/hbase/WALs/localhost,16203,1527647516612/localhost%2C16203%2C1527647516612.default.1527658324725.
BP-1128346793-127.0.0.1-1527647465315
blk_1073755283_14459{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-45a1fee3-50a1-44cc-8230-916d08f9b968:NORMAL:127.0.0.1:31001|RBW]]}
2018-05-29 22:32:04,740 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/hbase/WALs/localhost,16201,1527647513859/localhost%2C16201%2C1527647513859.default.1527658324731.
BP-1128346793-127.0.0.1-1527647465315
blk_1073755284_14460{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-223dfda7-738f-4020-bf4c-29a4642750e1:NORMAL:127.0.0.1:31002|RBW]]}
2018-05-29 22:32:04,748 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync:
/hbase/WALs/localhost,16203,1527647516612/localhost%2C16203%2C1527647516612.default.1527658324725
for DFSClient_NONMAPREDUCE_-454483218_1
2018-05-29 22:32:04,748 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync:
/hbase/WALs/localhost,16201,1527647513859/localhost%2C16201%2C1527647513859.default.1527658324731
for DFSClient_NONMAPREDUCE_1562981572_1
2018-05-29 22:32:04,755 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 127.0.0.1:31002 is added to
blk_1073755263_14439{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-223dfda7-738f-4020-bf4c-29a4642750e1:NORMAL:127.0.0.1:31002|RBW]]}
size 83
2018-05-29 22:32:04,755 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 127.0.0.1:31001 is added to
blk_1073755264_14440{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-45a1fee3-50a1-44cc-8230-916d08f9b968:NORMAL:127.0.0.1:31001|RBW]]}
size 83
2018-05-29 22:32:04,756 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/hbase/WALs/localhost,16203,1527647516612/localhost%2C16203%2C1527647516612.default.1527654724633
is closed by DFSClient_NONMAPREDUCE_-454483218_1
2018-05-29 22:32:04,756 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/hbase/WALs/localhost,16201,1527647513859/localhost%2C16201%2C1527647513859.default.1527654724642
is closed by DFSClient_NONMAPREDUCE_1562981572_1
2018-05-29 22:32:04,765 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/hbase/WALs/localhost,16202,1527647515279/localhost%2C16202%2C1527647515279.default.1527658324757.
BP-1128346793-127.0.0.1-1527647465315
blk_1073755285_14461{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-223dfda7-738f-4020-bf4c-29a4642750e1:NORMAL:127.0.0.1:31002|RBW]]}
2018-05-29 22:32:04,771 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync:
/hbase/WALs/localhost,16202,1527647515279/localhost%2C16202%2C1527647515279.default.1527658324757
for DFSClient_NONMAPREDUCE_-1203988066_1
2018-05-29 22:32:04,778 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated:
