[jira] [Updated] (IMPALA-8403) Possible thread leak in impalad
[
https://issues.apache.org/jira/browse/IMPALA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-8403:
---
Description:
The metric of thread-manager.running-threads got from
http://${impalad_host}:25000/metrics?json shows that the number of running
threads keeps increasing. (See the snapshot) This phenomenon is most noticeable
in coordinators.
Maybe a counter bug or threads leak.
was:
The metric of thread-manager.running-threads got from
http://${impalad_host}:25000/metrics?json shows that the number of running
threads keeps increasing. (See the snapshot)
Maybe a counter bug or threads leak.
> Possible thread leak in impalad
> ---
>
> Key: IMPALA-8403
> URL: https://issues.apache.org/jira/browse/IMPALA-8403
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: image-2019-04-10-11-15-11-321.png
>
>
> The metric of thread-manager.running-threads got from
> http://${impalad_host}:25000/metrics?json shows that the number of running
> threads keeps increasing. (See the snapshot) This phenomenon is most
> noticeable in coordinators.
> Maybe a counter bug or threads leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8403) Possible thread leak in impalad
Quanlong Huang created IMPALA-8403:
--
Summary: Possible thread leak in impalad
Key: IMPALA-8403
URL: https://issues.apache.org/jira/browse/IMPALA-8403
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Attachments: image-2019-04-10-11-15-11-321.png
The metric of thread-manager.running-threads got from
http://${impalad_host}:25000/metrics?json shows that the number of running
threads keeps increasing. (See the snapshot)
Maybe a counter bug or threads leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8370) Impala Doc: Impala works with Hive 3
[ https://issues.apache.org/jira/browse/IMPALA-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8370: Labels: future_release_doc in_33 (was: future_release_doc) > Impala Doc: Impala works with Hive 3 > > > Key: IMPALA-8370 > URL: https://issues.apache.org/jira/browse/IMPALA-8370 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8385) Refactor Sentry admin user check
[ https://issues.apache.org/jira/browse/IMPALA-8385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-8385. -- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Refactor Sentry admin user check > > > Key: IMPALA-8385 > URL: https://issues.apache.org/jira/browse/IMPALA-8385 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > Fix For: Impala 3.3.0 > > > Currently Sentry admin check is hardcoded, for example: > https://github.com/apache/impala/blob/5670f96b828d57f9e36510bb9af02bcc31de775c/be/src/service/client-request-state.cc#L366 > This check needs to be moved out to SentryAuthorizationManager instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8365) Add Ranger support with catalog v2 (local catalog) mode
[ https://issues.apache.org/jira/browse/IMPALA-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya reassigned IMPALA-8365: Assignee: Fredy Wijaya > Add Ranger support with catalog v2 (local catalog) mode > --- > > Key: IMPALA-8365 > URL: https://issues.apache.org/jira/browse/IMPALA-8365 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > > https://issues.apache.org/jira/browse/IMPALA-8100 only works with catalog v1. > We need to add support for Ranger with local catalog (v2) mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8364) Impala Doc: Remove support for authorization policy file
[ https://issues.apache.org/jira/browse/IMPALA-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813911#comment-16813911 ] Alex Rodoni commented on IMPALA-8364: - Target version has not been decided for this feature. > Impala Doc: Remove support for authorization policy file > > > Key: IMPALA-8364 > URL: https://issues.apache.org/jira/browse/IMPALA-8364 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8364) Impala Doc: Remove support for authorization policy file
[ https://issues.apache.org/jira/browse/IMPALA-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8364: Target Version: (was: Impala 3.3.0) > Impala Doc: Remove support for authorization policy file > > > Key: IMPALA-8364 > URL: https://issues.apache.org/jira/browse/IMPALA-8364 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8364) Impala Doc: Remove support for authorization policy file
[ https://issues.apache.org/jira/browse/IMPALA-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8364: Labels: future_release_doc (was: future_release_doc in_33) > Impala Doc: Remove support for authorization policy file > > > Key: IMPALA-8364 > URL: https://issues.apache.org/jira/browse/IMPALA-8364 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7878) Bad SQL generated by compute incremental stats
[
https://issues.apache.org/jira/browse/IMPALA-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fang-Yu Rao reassigned IMPALA-7878:
---
Assignee: Fang-Yu Rao
> Bad SQL generated by compute incremental stats
> ---
>
> Key: IMPALA-7878
> URL: https://issues.apache.org/jira/browse/IMPALA-7878
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Reporter: Pooja Nilangekar
>Assignee: Fang-Yu Rao
>Priority: Major
>
> Computing incremental stats on partitions generates bad sql for instance:
> For a table foo partitioned by column bar, the compute stats statement:
> {code:java}
> compute incremental stats foo partition (bar = 1);
> {code}
> would generate the following query:
> {code:java}
> SELECT COUNT(*), month FROM foo WHERE (bar=1) GROUP BY bar;
> {code}
> If this were to be rewritten as follows, it would produce fewer fragments and
> hence also reduce query memory by avoiding a hash aggregation node.
> {code:java}
> SELECT COUNT(*), 1 FROM foo WHERE bar=1;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8391) Impala Doc: ALTER TABLE RENAME on managed Kudu table renames underlying Kudu table
[
https://issues.apache.org/jira/browse/IMPALA-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8391 started by Alex Rodoni.
---
> Impala Doc: ALTER TABLE RENAME on managed Kudu table renames underlying Kudu
> table
> --
>
> Key: IMPALA-8391
> URL: https://issues.apache.org/jira/browse/IMPALA-8391
> Project: IMPALA
> Issue Type: Sub-task
> Components: Docs
>Reporter: Sahil Takiar
>Assignee: Alex Rodoni
>Priority: Major
> Labels: future_release_doc, in_33
>
> The Impala-Kudu docs:
> [http://impala.apache.org/docs/build/html/topics/impala_kudu.html]
> [http://impala.apache.org/docs/build/html/topics/impala_tables.html]
> Need to be updated after IMPALA-7640 is merged.
> Specifically this part of the docs will no longer be accurate:
> {quote}
> When you create a Kudu table through Impala, it is assigned an internal Kudu
> table name of the form {{impala::db_name.table_name}}. You can see the
> Kudu-assigned name in the output of {{DESCRIBE FORMATTED}}, in the
> {{kudu.table_name}} field of the table properties. The Kudu-assigned name
> remains the same even if you use {{ALTER TABLE}} to rename the Impala table
> or move it to a different Impala database. You can issue the statement{{ALTER
> TABLE impala_name SET TBLPROPERTIES('kudu.table_name' =
> 'different_kudu_table_name')}} for the external tables created with the
> {{CREATE EXTERNAL TABLE}} statement. Changing the {{kudu.table_name}}property
> of an external table switches which underlying Kudu table the Impala table
> refers to. The underlying Kudu table must already exist.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8391) Impala Doc: ALTER TABLE RENAME on managed Kudu table renames underlying Kudu table
[
https://issues.apache.org/jira/browse/IMPALA-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813897#comment-16813897
]
Alex Rodoni commented on IMPALA-8391:
-
https://gerrit.cloudera.org/#/c/12975/
> Impala Doc: ALTER TABLE RENAME on managed Kudu table renames underlying Kudu
> table
> --
>
> Key: IMPALA-8391
> URL: https://issues.apache.org/jira/browse/IMPALA-8391
> Project: IMPALA
> Issue Type: Sub-task
> Components: Docs
>Reporter: Sahil Takiar
>Assignee: Alex Rodoni
>Priority: Major
> Labels: future_release_doc, in_33
>
> The Impala-Kudu docs:
> [http://impala.apache.org/docs/build/html/topics/impala_kudu.html]
> [http://impala.apache.org/docs/build/html/topics/impala_tables.html]
> Need to be updated after IMPALA-7640 is merged.
> Specifically this part of the docs will no longer be accurate:
> {quote}
> When you create a Kudu table through Impala, it is assigned an internal Kudu
> table name of the form {{impala::db_name.table_name}}. You can see the
> Kudu-assigned name in the output of {{DESCRIBE FORMATTED}}, in the
> {{kudu.table_name}} field of the table properties. The Kudu-assigned name
> remains the same even if you use {{ALTER TABLE}} to rename the Impala table
> or move it to a different Impala database. You can issue the statement{{ALTER
> TABLE impala_name SET TBLPROPERTIES('kudu.table_name' =
> 'different_kudu_table_name')}} for the external tables created with the
> {{CREATE EXTERNAL TABLE}} statement. Changing the {{kudu.table_name}}property
> of an external table switches which underlying Kudu table the Impala table
> refers to. The underlying Kudu table must already exist.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8402) Add targeted perf tests for CHAR
Tim Armstrong created IMPALA-8402: - Summary: Add targeted perf tests for CHAR Key: IMPALA-8402 URL: https://issues.apache.org/jira/browse/IMPALA-8402 Project: IMPALA Issue Type: Improvement Components: Infrastructure Reporter: Tim Armstrong This is follow-on from IMPALA-7331. We don't currently have targeted perf coverage for CHAR as a result of not having any CHAR-type columns in TPC-H or TPC-DS. CHAR is not a preferred data tyandpe in Impala because of the "interesting" space-padding semantics and limited codegen support but people do use it in production. It would be nice to have a way to measure perf regressions or take credit for perf wins if/when we enabled codegen for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8312) Alter database operations have race condition
[
https://issues.apache.org/jira/browse/IMPALA-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar resolved IMPALA-8312.
-
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Alter database operations have race condition
> -
>
> Key: IMPALA-8312
> URL: https://issues.apache.org/jira/browse/IMPALA-8312
> Project: IMPALA
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: Impala 3.3.0
>
>
> There are currently two {{alter database}} operations seen in
> {{CatalogOpExecutor}} although I only see one of them documented which is a
> separate issue.
> Consider the following snippet for {{alter database set owner}} operation.
> {code}
> private void alterDatabaseSetOwner(String dbName, TAlterDbSetOwnerParams
> params,
> TDdlExecResponse response) throws ImpalaException {
> Db db = catalog_.getDb(dbName);
> if (db == null) {
> throw new CatalogException("Database: " + dbName + " does not exist.");
> }
> Preconditions.checkNotNull(params.owner_name);
> Preconditions.checkNotNull(params.owner_type);
> synchronized (metastoreDdlLock_) {
> Database msDb = db.getMetaStoreDb();
> String originalOwnerName = msDb.getOwnerName();
> PrincipalType originalOwnerType = msDb.getOwnerType();
> msDb.setOwnerName(params.owner_name);
> msDb.setOwnerType(PrincipalType.valueOf(params.owner_type.name()));
> try {
> applyAlterDatabase(db);
> } catch (ImpalaRuntimeException e) {
> msDb.setOwnerName(originalOwnerName);
> msDb.setOwnerType(originalOwnerType);
> throw e;
> }
> updateOwnerPrivileges(db.getName(), /* tableName */ null,
> params.server_name,
> originalOwnerName, originalOwnerType,
> db.getMetaStoreDb().getOwnerName(),
> db.getMetaStoreDb().getOwnerType(), response);
> }
> addDbToCatalogUpdate(db, response.result);
> addSummary(response, "Updated database.");
> }
> {code}
> If you notice above, it takes a lock on {{metastoreDdlLock_}} but does not
> take a write lock on catalogVersion before altering the metastore db object
> in-place. This can lead to race conditions between a reader and writer
> thread. For example, it is possible that the thread which is reading {{msDb}}
> object can see new value of owner but a old value of owner type.
> The same problem applies to the alterCommentOnDb method below
> {code}
> private void alterCommentOnDb(String dbName, String comment,
> TDdlExecResponse response)
> throws ImpalaRuntimeException, CatalogException {
> Db db = catalog_.getDb(dbName);
> if (db == null) {
> throw new CatalogException("Database: " + dbName + " does not exist.");
> }
> synchronized (metastoreDdlLock_) {
> Database msDb = db.getMetaStoreDb();
> String originalComment = msDb.getDescription();
> msDb.setDescription(comment);
> try {
> applyAlterDatabase(db);
> } catch (ImpalaRuntimeException e) {
> msDb.setDescription(originalComment);
> throw e;
> }
> }
> addDbToCatalogUpdate(db, response.result);
> addSummary(response, "Updated database.");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8393) setup-ranger step in create-load-data.sh breaks data load to real clusters
[
https://issues.apache.org/jira/browse/IMPALA-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Austin Nobis resolved IMPALA-8393.
--
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> setup-ranger step in create-load-data.sh breaks data load to real clusters
> --
>
> Key: IMPALA-8393
> URL: https://issues.apache.org/jira/browse/IMPALA-8393
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0, Impala 3.3.0
>Reporter: David Knupp
>Assignee: Austin Nobis
>Priority: Critical
> Fix For: Impala 3.3.0
>
>
> {{localhost}} is hard-coded into the setup-ranger function that was recently
> added to create-load-data.sh, e.g.:
> https://github.com/apache/impala/blame/master/testdata/bin/create-load-data.sh#L325
> This works when testing on a mini-cluster, but breaks data load if setting up
> to run the functional test suite against an actual cluster. In that scenario,
> the host that runs the script is simply a test runner, with no locally
> running services.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5351) Handle column comments for Kudu tables
[ https://issues.apache.org/jira/browse/IMPALA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall reassigned IMPALA-5351: -- Assignee: HeLifu > Handle column comments for Kudu tables > -- > > Key: IMPALA-5351 > URL: https://issues.apache.org/jira/browse/IMPALA-5351 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.9.0 >Reporter: Thomas Tauber-Marshall >Assignee: HeLifu >Priority: Major > Labels: kudu > > Currently, if a column comment is specified in a CREATE TABLE for a Kudu > table, we just silently drop it because Kudu does not currently support > column comments (KUDU-1711). > One option would be to store the comments in HMS, but splitting the metadata > between HMS and Kudu is probably more complicated than its worth. > Most likely, we can just wait for it to be implemented on the Kudu side, but > before then we may want to consider issuing a warning when people use column > comments on Kudu tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5351) Handle column comments for Kudu tables
[ https://issues.apache.org/jira/browse/IMPALA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813646#comment-16813646 ] Grant Henke commented on IMPALA-5351: - Sounds good [~helifu]. > Handle column comments for Kudu tables > -- > > Key: IMPALA-5351 > URL: https://issues.apache.org/jira/browse/IMPALA-5351 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.9.0 >Reporter: Thomas Tauber-Marshall >Priority: Major > Labels: kudu > > Currently, if a column comment is specified in a CREATE TABLE for a Kudu > table, we just silently drop it because Kudu does not currently support > column comments (KUDU-1711). > One option would be to store the comments in HMS, but splitting the metadata > between HMS and Kudu is probably more complicated than its worth. > Most likely, we can just wait for it to be implemented on the Kudu side, but > before then we may want to consider issuing a warning when people use column > comments on Kudu tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8336) TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
[
https://issues.apache.org/jira/browse/IMPALA-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813557#comment-16813557
]
ASF subversion and git services commented on IMPALA-8336:
-
Commit ead4d4d1dfc165f014f4bad331920e4e951e3fcb in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ead4d4d ]
IMPALA-8336: fix flaky ORC memory limit test
Reduces the mem_limit for the ORC version of the
test, which has proven to be flaky.
Testing:
Looped the test for a while locally without any failures.
I was unable to reproduce the failure seen on CentOS6 jenkins locally,
so we'll just try this tweak and see if it improves this. If not
I can look deeper into it.
Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7
Reviewed-on: http://gerrit.cloudera.org:8080/12961
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
> -
>
> Key: IMPALA-8336
> URL: https://issues.apache.org/jira/browse/IMPALA-8336
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
>
> The test expects the memory limit to be exceeded, but it doesn't happen:
> {noformat}
> query_test.test_nested_types.TestNestedTypes.test_tpch_mem_limit_single_node[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> orc/def/block]
> query_test/test_nested_types.py:123: in test_tpch_mem_limit_single_node
> new_vector, use_db='tpch_nested' + db_suffix)
> common/impala_test_suite.py:487: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex: .*Memory limit exceeded:
> Failed to allocate [0-9]+ bytes for collection
> 'tpch_nested_.*.customer.c_orders.item.o_lineitems'.*{noformat}
> Seen once on Centos 6.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5051) Add support to write INT64 timestamps to the parquet writer
[ https://issues.apache.org/jira/browse/IMPALA-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813556#comment-16813556 ] ASF subversion and git services commented on IMPALA-5051: - Commit 39413a18117acde1822d9f084ab30c748ce837bc in impala's branch refs/heads/master from Csaba Ringhofer [ https://gitbox.apache.org/repos/asf?p=impala.git;h=39413a1 ] IMPALA-5051: Add INT64 timestamp write support in Parquet Add query option "parquet_timestamp_type" that chooses the Parquet type used when writing TIMESTAMP columns. This is an experimental feature at the moment, because these types are not widely adopted in other Hadoop components yet. For this reason the query option is added as "development" level, and the default behavior is not changed. The following options can be used: INT96_NANOS (default): This is the same as the old behavior, can represent any timestamp that can be handled by Impala. INT64_MILLIS, INT64_MICROS: Can encode the whole [1400..1) range handled by Impala at the cost of reduced precision. Values are rounded towards minus infinity during writing. INT64_NANOS: Can encode a reduced range without losing nanosecond precision: [1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807] Values outside this range are converted to NULLs without warning. The change was done completely in the backend and all TIMESTAMP columns are written using the type set in the query option. An alternative design would have been to implement some parts in the fronted by adding TIMESTAMP->BIGINT conversion functions to the query plan, which would make it easier to add the possibility of per-column setting in the future. I choose the current design because it seemed much simpler and there are no clear plans for the per-column setting. Most of the code will be still useful if we decide to go the other way in the future. All types are written without conversion to UTC (the way Impala always wrote timestamps), and this information is expressed in the new Parquet logical types by setting isAdjustedToUTC to false. The old logical type (converted_type) is not set, because old readers do not read isAdjustedToUTC, and assume that TIMESTAMP_MILLIS and TIMESTAMP_MICROS are written in UTC. These readers can still read int64 timestamp columns as INT_64. Testing: - added unit tests for new TimestampValue->int64 functions - add EE tests for checking values / min-max stats / metadata written for int64 Parquet timestamps - ran core tests Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Reviewed-on: http://gerrit.cloudera.org:8080/12247 Tested-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer > Add support to write INT64 timestamps to the parquet writer > --- > > Key: IMPALA-5051 > URL: https://issues.apache.org/jira/browse/IMPALA-5051 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 2.9.0 >Reporter: Lars Volker >Assignee: Csaba Ringhofer >Priority: Major > > This requires updating parquet.thrift to a version that includes the > TIMESTAMP_MICROS logical type. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8371) Unified backend tests need to return appropriate return code
[ https://issues.apache.org/jira/browse/IMPALA-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813559#comment-16813559 ] ASF subversion and git services commented on IMPALA-8371: - Commit 45364a47c12a71eee9ee5592e05a8fc2295951d5 in impala's branch refs/heads/master from Joe McDonnell [ https://gitbox.apache.org/repos/asf?p=impala.git;h=45364a4 ] IMPALA-8371: Return appropriate error code for unified backend tests Unified backend tests rely on generating bash scripts for each test that call the unified executable with a filter to run the appropriate subset of the tests. The generated script currently does not return the return code from the test execution. This changes the test execution scripts to return the appropriate return code. To do this, the script generator is changed from a bash implementation in bin/gen-backend-test-script.sh to a python implementation in bin/gen-backend-test-script.py. This makes it easier to handle shell variables in the script template correctly. Testing: - Ran backend tests on centos 6, centos 7 - Manually tested with a failing test and verified return value Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Reviewed-on: http://gerrit.cloudera.org:8080/12885 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins > Unified backend tests need to return appropriate return code > > > Key: IMPALA-8371 > URL: https://issues.apache.org/jira/browse/IMPALA-8371 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Blocker > Labels: broken-build > > The scripts generated by bin/gen-backend-test-script.sh need to return the > return code from the call to the unified backend executable. The JUnitXML > contains a failure, which Jenkins and other tools can process, but the return > code must match up for scripts to be able to loop the test, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8395) SystemStateInfoTest.ReadProcNetDev failing on Centos 6
[
https://issues.apache.org/jira/browse/IMPALA-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813558#comment-16813558
]
ASF subversion and git services commented on IMPALA-8395:
-
Commit d808bcda9df74d02d452bfa9f4ce491dd873f241 in impala's branch
refs/heads/master from Lars Volker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d808bcd ]
IMPALA-8395: Parse older formats of /proc/net/dev correctly
Older kernel versions don't have a space between the interface name and
the first counter value in /proc/net/dev. This change reworks the
parsing logic to support such older formats and adds a unit test for it.
Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426
Reviewed-on: http://gerrit.cloudera.org:8080/12954
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> SystemStateInfoTest.ReadProcNetDev failing on Centos 6
> --
>
> Key: IMPALA-8395
> URL: https://issues.apache.org/jira/browse/IMPALA-8395
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Joe McDonnell
>Assignee: Lars Volker
>Priority: Blocker
> Labels: broken-build
>
> On Centos 6, SystemStateInfoTest is failing:
> {noformat}
> Error Message
> Expected: (state[SystemStateInfo::NET_RX_PACKETS]) > (0), actual: 0 vs 0
> Stacktrace
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/be/src/util/system-state-info-test.cc:55
> Expected: (state[SystemStateInfo::NET_RX_PACKETS]) > (0), actual: 0 vs
> 0{noformat}
> Unfortunately, this is being masked by IMPALA-8371, which prevents this from
> reporting the test as failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
