[jira] [Created] (IMPALA-8248) Re-organize authorization tests

2019-02-25 Thread Fredy Wijaya (JIRA)
Fredy Wijaya created IMPALA-8248:


 Summary: Re-organize authorization tests
 Key: IMPALA-8248
 URL: https://issues.apache.org/jira/browse/IMPALA-8248
 Project: IMPALA
  Issue Type: Sub-task
  Components: Infrastructure
Reporter: Fredy Wijaya


We have authorization tests that are specific to Sentry and authorization tests 
that can be applicable to any authorization provider. We need to re-organize 
the authorization tests to easily differentiate between Sentry-specific tests 
vs generic authorization tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (IMPALA-8248) Re-organize authorization tests

2019-02-25 Thread Fredy Wijaya (JIRA)
Fredy Wijaya created IMPALA-8248:


 Summary: Re-organize authorization tests
 Key: IMPALA-8248
 URL: https://issues.apache.org/jira/browse/IMPALA-8248
 Project: IMPALA
  Issue Type: Sub-task
  Components: Infrastructure
Reporter: Fredy Wijaya


We have authorization tests that are specific to Sentry and authorization tests 
that can be applicable to any authorization provider. We need to re-organize 
the authorization tests to easily differentiate between Sentry-specific tests 
vs generic authorization tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-7916) Support for Apache Ranger authorization provider

2019-02-25 Thread Fredy Wijaya (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fredy Wijaya updated IMPALA-7916:
-
Epic Name: ranger  (was: authz-provider)

> Support for Apache Ranger authorization provider
> 
>
> Key: IMPALA-7916
> URL: https://issues.apache.org/jira/browse/IMPALA-7916
> Project: IMPALA
>  Issue Type: Epic
>  Components: Catalog, Frontend
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Major
> Fix For: Impala 3.3.0
>
>
> Currently Impala has no easy way to support other authorization providers 
> since the code is very tightly coupled to Sentry. The purpose of this ticket 
> is to provide support for pluggable authorization provider in Impala. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-8243) ConcurrentModificationException in Catalog stress tests

2019-02-25 Thread Adrian Ng (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrian Ng updated IMPALA-8243:
--
Target Version: Impala 3.2.0

> ConcurrentModificationException in Catalog stress tests
> ---
>
> Key: IMPALA-8243
> URL: https://issues.apache.org/jira/browse/IMPALA-8243
> Project: IMPALA
>  Issue Type: Bug
>  Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: bharath v
>Priority: Blocker
>
> Following is the full stack from the Catalog server logs.
> {noformat}
> 14:09:29.474424 14829 jni-util.cc:256] 
> java.util.ConcurrentModificationException
> java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> java.util.ArrayList$Itr.next(ArrayList.java:851)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1449)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1278)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1144)
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1062)
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:919)
> org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:815)
> org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:862)
> org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:759)
> org.apache.impala.thrift.TPartialPartitionInfo.write(TPartialPartitionInfo.java:665)
> org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:731)
> org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:624)
> org.apache.impala.thrift.TPartialTableInfo.write(TPartialTableInfo.java:543)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:977)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:857)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse.write(TGetPartialCatalogObjectResponse.java:739)
> org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:233)
> {noformat}
> It looks like the bug is in the following piece of code.
> {noformat}
> /**
>* Returns a Hive-compatible partition object that may be used in calls to 
> the
>* metastore.
>*/
>   public org.apache.hadoop.hive.metastore.api.Partition toHmsPartition() {
> if (cachedMsPartitionDescriptor_ == null) return null;
> Preconditions.checkNotNull(table_.getNonPartitionFieldSchemas());
> // Update the serde library class based on the currently used file format.
> org.apache.hadoop.hive.metastore.api.StorageDescriptor storageDescriptor =
> new org.apache.hadoop.hive.metastore.api.StorageDescriptor(
> table_.getNonPartitionFieldSchemas(),  <= Reference to the 
> actual field schema list.
> getLocation(),
> cachedMsPartitionDescriptor_.sdInputFormat,
> cachedMsPartitionDescriptor_.sdOutputFormat,
> cachedMsPartitionDescriptor_.sdCompressed,
> {noformat}
> It appears we are leaking a reference to {{nonPartFieldSchemas_}} in to the 
> thrift object and once the thread leaves the lock scope, some other thread 
> (load() for ex: ) can potentially change the source list and the 
> serialization code could throw {{ConcurrentModificationException}}
> While the stack above is Catalog-v2 only, it is possible that some other 
> threads can race in a similar fashion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-8243) ConcurrentModificationException in Catalog stress tests

2019-02-25 Thread Adrian Ng (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrian Ng updated IMPALA-8243:
--
Priority: Blocker  (was: Major)

> ConcurrentModificationException in Catalog stress tests
> ---
>
> Key: IMPALA-8243
> URL: https://issues.apache.org/jira/browse/IMPALA-8243
> Project: IMPALA
>  Issue Type: Bug
>  Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: bharath v
>Priority: Blocker
>
> Following is the full stack from the Catalog server logs.
> {noformat}
> 14:09:29.474424 14829 jni-util.cc:256] 
> java.util.ConcurrentModificationException
> java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> java.util.ArrayList$Itr.next(ArrayList.java:851)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1449)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1278)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1144)
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1062)
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:919)
> org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:815)
> org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:862)
> org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:759)
> org.apache.impala.thrift.TPartialPartitionInfo.write(TPartialPartitionInfo.java:665)
> org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:731)
> org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:624)
> org.apache.impala.thrift.TPartialTableInfo.write(TPartialTableInfo.java:543)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:977)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:857)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse.write(TGetPartialCatalogObjectResponse.java:739)
> org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:233)
> {noformat}
> It looks like the bug is in the following piece of code.
> {noformat}
> /**
>* Returns a Hive-compatible partition object that may be used in calls to 
> the
>* metastore.
>*/
>   public org.apache.hadoop.hive.metastore.api.Partition toHmsPartition() {
> if (cachedMsPartitionDescriptor_ == null) return null;
> Preconditions.checkNotNull(table_.getNonPartitionFieldSchemas());
> // Update the serde library class based on the currently used file format.
> org.apache.hadoop.hive.metastore.api.StorageDescriptor storageDescriptor =
> new org.apache.hadoop.hive.metastore.api.StorageDescriptor(
> table_.getNonPartitionFieldSchemas(),  <= Reference to the 
> actual field schema list.
> getLocation(),
> cachedMsPartitionDescriptor_.sdInputFormat,
> cachedMsPartitionDescriptor_.sdOutputFormat,
> cachedMsPartitionDescriptor_.sdCompressed,
> {noformat}
> It appears we are leaking a reference to {{nonPartFieldSchemas_}} in to the 
> thrift object and once the thread leaves the lock scope, some other thread 
> (load() for ex: ) can potentially change the source list and the 
> serialization code could throw {{ConcurrentModificationException}}
> While the stack above is Catalog-v2 only, it is possible that some other 
> threads can race in a similar fashion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-8239) Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0

2019-02-25 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777587#comment-16777587
 ] 

ASF subversion and git services commented on IMPALA-8239:
-

Commit 1718f9c07bfac2e14d0d9a2de052dea8ac65f45f in impala's branch 
refs/heads/master from Michael Ho
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1718f9c ]

IMPALA-8239: Fix handling of failed deserialization of row batch

Previously, when a row batch failed to be deserialized in the
data stream receiver, we will return the error status to the
sender of the row batch without inserting the row batch. The
receiver will continue to operate without flagging any error.
The assumption is that the sender will eventually cancel the
query upon receiving the failed status.

Normally, when a caller of GetBatch() successfully dequeues a row
batch from the batch queue, it will kick off the draining of the
row batches from the deferred queue into the normal batch queue,
which will further continue the cycle of draining the deferred queue
upon the next call to GetBatch() until the deferred queue becomes empty.

When an error is hit when deserializing a deferred batch to be inserted
into the batch queue, the existing code will simply not insert the row
batch or flag any error. This breaks the cycle of the deferred queue
draining as the batch queue may become empty forever. The caller of
GetBatch() will block indefinitely until the query is cancelled.

The existing code works fine as the expectation is that the query will
be cancelled once the sender receives the error status from the RPC
response. However, this behavior is still not ideal as it lets a query
which has hit a fatal error to hold on to resources for extended period
of time.

This patch fixes the problem by explicitly recording any error during
row batch insertion in an error status object. Callers of GetBatch()
will now also poll for this status object while waiting for row batch
to show up and bail out early if there is any error.

A new test case has been added to simulate the problematic case above.

Change-Id: Iaa74937b046d95484887533be548249e96078755
Reviewed-on: http://gerrit.cloudera.org:8080/12567
Reviewed-by: Tim Armstrong 
Reviewed-by: Thomas Marshall 
Tested-by: Impala Public Jenkins 


>  Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + 
> num_pending_enqueue_) > 0
> 
>
> Key: IMPALA-8239
> URL: https://issues.apache.org/jira/browse/IMPALA-8239
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Blocker
>  Labels: crash
> Attachments: IMPALA-8239-logs.tar.gz
>
>
> {noformat}
> F0221 12:36:27.886497 157443 krpc-data-stream-recvr.cc:234] 
> 3843237c92f4c6b5:b5aa170b0092] Check failed: deferred_rpcs_.empty() || 
> (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0 
> *** Check failure stack trace: ***
> @  0x47eb92c  google::LogMessage::Fail()
> @  0x47ed1d1  google::LogMessage::SendToLog()
> @  0x47eb306  google::LogMessage::Flush()
> @  0x47ee8cd  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ed07a0  impala::KrpcDataStreamRecvr::SenderQueue::GetBatch()
> @  0x1ed65e7  impala::KrpcDataStreamRecvr::GetBatch()
> @  0x22c5d67  impala::ExchangeNode::FillInputRowBatch()
> @  0x22c5953  impala::ExchangeNode::Open()
> @  0x240079a  
> impala::BlockingJoinNode::ProcessBuildInputAndOpenProbe()
> @  0x238720f  impala::PartitionedHashJoinNode::Open()
> @  0x23e6d76  impala::AggregationNode::Open()
> @  0x1f52b2f  impala::FragmentInstanceState::Open()
> @  0x1f4f7f3  impala::FragmentInstanceState::Exec()
> @  0x1f62dd9  impala::QueryState::ExecFInstance()
> @  0x1f610bb  
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @  0x1f6421a  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1d766b3  boost::function0<>::operator()()
> @  0x2223e4e  impala::Thread::SuperviseThread()
> @  0x222c1d2  boost::_bi::list5<>::operator()<>()
> @  0x222c0f6  boost::_bi::bind_t<>::operator()()
> @  0x222c0b9  boost::detail::thread_data<>::run()
> @  0x3715469  thread_proxy
> @ 0x7f262e1efe24  start_thread
> @ 0x7f262df1d34c  __clone
> {noformat}
> Immediately before it in the log a different thread printed:
> {noformat}
> E0221 12:36:27.557099 157280 

[jira] [Commented] (IMPALA-8154) Disable auth_to_local by default

2019-02-25 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777586#comment-16777586
 ] 

ASF subversion and git services commented on IMPALA-8154:
-

Commit 63eee03968c78fafd8afa23cf3f8c93ad415d796 in impala's branch 
refs/heads/master from Alex Rodoni
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=63eee03 ]

[DOCS] Noted IMPALA-8154 as fixed in Known Issues doc

Change-Id: I4ecae87938cc2ee9ce5a2bed639d3719f9bd793e
Reviewed-on: http://gerrit.cloudera.org:8080/12551
Tested-by: Impala Public Jenkins 
Reviewed-by: Michael Ho 


> Disable auth_to_local by default
> 
>
> Key: IMPALA-8154
> URL: https://issues.apache.org/jira/browse/IMPALA-8154
> Project: IMPALA
>  Issue Type: Bug
>  Components: Distributed Exec
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Major
> Fix For: Impala 3.2.0
>
>
> Before KRPC the local name mapping was done from the principal name entirely, 
> however when KRPC is enabled Impala starts to use the system auth_to_local 
> rules, "use_system_auth_to_local" is enabled by default. This can cause 
> regression in cases where localauth is configured in the krb5.conf. This may 
> cause issue for connection between Impalad after [this 
> commit|https://github.com/apache/impala/commit/5c541b960491ba91533712144599fb3b6d99521d]
> The fix is to disable use_system_auth_to_local by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-8243) ConcurrentModificationException in Catalog stress tests

2019-02-25 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777588#comment-16777588
 ] 

ASF subversion and git services commented on IMPALA-8243:
-

Commit fced1cc1bbe62bb94b82962aea0560df8ed00d2d in impala's branch 
refs/heads/master from Bharath Vissapragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fced1cc ]

IMPALA-8243: Fix racy access to nonPartFieldSchemas_

** Please refer to the jira for the full stacktrace.

When constructing the HMS partition state for a given partition,
we are leaking a reference to HdfsTable#nonPartFieldSchemas_ list.
The construction happens under a table lock. But once the thread
exits the lock scope, the source list could be racily modified by
another operation (say refresh) and that interferes with the original
thread if it tries to access the list.`

The fix is to make a shallow copy of the source list so that any
changes to the list do not affect the original caller.

This was found by a stress test under heavy concurrency of refresh
operations + GetPartialCatalogObject() calls.

Testing:
-
- I tried a bunch of combinations of operations in the unit-test
framework but I couldn't reproduce the stack trace, probably because
the operations are very short-lived

- However, after deploying this patched jar on the stress test cluster,
this exception never happened again.

Change-Id: I7d68b54af2ba954cf0ffa7b2533cde7be835be77
Reviewed-on: http://gerrit.cloudera.org:8080/12572
Tested-by: Impala Public Jenkins 
Reviewed-by: Paul Rogers 


> ConcurrentModificationException in Catalog stress tests
> ---
>
> Key: IMPALA-8243
> URL: https://issues.apache.org/jira/browse/IMPALA-8243
> Project: IMPALA
>  Issue Type: Bug
>  Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: bharath v
>Priority: Major
>
> Following is the full stack from the Catalog server logs.
> {noformat}
> 14:09:29.474424 14829 jni-util.cc:256] 
> java.util.ConcurrentModificationException
> java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> java.util.ArrayList$Itr.next(ArrayList.java:851)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1449)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1278)
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1144)
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1062)
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:919)
> org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:815)
> org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:862)
> org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:759)
> org.apache.impala.thrift.TPartialPartitionInfo.write(TPartialPartitionInfo.java:665)
> org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:731)
> org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:624)
> org.apache.impala.thrift.TPartialTableInfo.write(TPartialTableInfo.java:543)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:977)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:857)
> org.apache.impala.thrift.TGetPartialCatalogObjectResponse.write(TGetPartialCatalogObjectResponse.java:739)
> org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
> org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:233)
> {noformat}
> It looks like the bug is in the following piece of code.
> {noformat}
> /**
>* Returns a Hive-compatible partition object that may be used in calls to 
> the
>* metastore.
>*/
>   public org.apache.hadoop.hive.metastore.api.Partition toHmsPartition() {
> if (cachedMsPartitionDescriptor_ == null) return null;
> Preconditions.checkNotNull(table_.getNonPartitionFieldSchemas());
> // Update the serde library class based on the currently used file format.
> org.apache.hadoop.hive.metastore.api.StorageDescriptor storageDescriptor =
> new org.apache.hadoop.hive.metastore.api.StorageDescriptor(
> table_.getNonPartitionFieldSchemas(),  <= Reference to the 
> actual field schema list.
> getLocation(),
> 

[jira] [Commented] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running

2019-02-25 Thread Joe McDonnell (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777511#comment-16777511
 ] 

Joe McDonnell commented on IMPALA-8247:
---

Code review:

https://gerrit.cloudera.org/#/c/12584/

> Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are 
> not running
> -
>
> Key: IMPALA-8247
> URL: https://issues.apache.org/jira/browse/IMPALA-8247
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend, Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Blocker
>  Labels: broken-build
>
> Several tests in be/src/util were converted to the unified backend test 
> executable in IMPALA-8071. Unfortunately, some tests stopped running as part 
> of this change. Specifically, the tests in bit-stream-utils-test.cc and 
> system-state-info-test.cc were not included in the test binary and stopped 
> running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files 
> were not included in the UtilTests library.
> These two files should be included in the test executable and we need to 
> update the validation of the unified test executable 
> (bin/validate-unified-backend-test-filters.py) to catch this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Created] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running

2019-02-25 Thread Joe McDonnell (JIRA)
Joe McDonnell created IMPALA-8247:
-

 Summary: Backend tests from bit-stream-utils-test.cc and 
system-state-info-test.cc are not running
 Key: IMPALA-8247
 URL: https://issues.apache.org/jira/browse/IMPALA-8247
 Project: IMPALA
  Issue Type: Bug
  Components: Backend, Infrastructure
Affects Versions: Impala 3.2.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell


Several tests in be/src/util were converted to the unified backend test 
executable in IMPALA-8071. Unfortunately, some tests stopped running as part of 
this change. Specifically, the tests in bit-stream-utils-test.cc and 
system-state-info-test.cc were not included in the test binary and stopped 
running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files were 
not included in the UtilTests library.

These two files should be included in the test executable and we need to update 
the validation of the unified test executable 
(bin/validate-unified-backend-test-filters.py) to catch this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running

2019-02-25 Thread Joe McDonnell (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777492#comment-16777492
 ] 

Joe McDonnell commented on IMPALA-8247:
---

I have a code change to fix this. I am testing it now.

> Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are 
> not running
> -
>
> Key: IMPALA-8247
> URL: https://issues.apache.org/jira/browse/IMPALA-8247
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend, Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Blocker
>  Labels: broken-build
>
> Several tests in be/src/util were converted to the unified backend test 
> executable in IMPALA-8071. Unfortunately, some tests stopped running as part 
> of this change. Specifically, the tests in bit-stream-utils-test.cc and 
> system-state-info-test.cc were not included in the test binary and stopped 
> running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files 
> were not included in the UtilTests library.
> These two files should be included in the test executable and we need to 
> update the validation of the unified test executable 
> (bin/validate-unified-backend-test-filters.py) to catch this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-7974) Impala Doc: Doc the options to enable automatic invalidates using metastore notification events

2019-02-25 Thread Alex Rodoni (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni updated IMPALA-7974:

Summary: Impala Doc: Doc the options to enable automatic invalidates using 
metastore notification events  (was: Impala Doc: Doc automatic invalidates 
using metastore notification events)

> Impala Doc: Doc the options to enable automatic invalidates using metastore 
> notification events
> ---
>
> Key: IMPALA-7974
> URL: https://issues.apache.org/jira/browse/IMPALA-7974
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>  Labels: future_release_doc, in_32
>
> https://gerrit.cloudera.org/#/c/12549/
> https://gerrit.cloudera.org/#/c/12118/
> https://gerrit.cloudera.org/#/c/12365/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Work started] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running

2019-02-25 Thread Joe McDonnell (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on IMPALA-8247 started by Joe McDonnell.
-
> Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are 
> not running
> -
>
> Key: IMPALA-8247
> URL: https://issues.apache.org/jira/browse/IMPALA-8247
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend, Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Blocker
>  Labels: broken-build
>
> Several tests in be/src/util were converted to the unified backend test 
> executable in IMPALA-8071. Unfortunately, some tests stopped running as part 
> of this change. Specifically, the tests in bit-stream-utils-test.cc and 
> system-state-info-test.cc were not included in the test binary and stopped 
> running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files 
> were not included in the UtilTests library.
> These two files should be included in the test executable and we need to 
> update the validation of the unified test executable 
> (bin/validate-unified-backend-test-filters.py) to catch this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-7974) Impala Doc: Doc automatic invalidates using metastore notification events

2019-02-25 Thread Alex Rodoni (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni updated IMPALA-7974:

Description: 
https://gerrit.cloudera.org/#/c/12549/
https://gerrit.cloudera.org/#/c/12118/
https://gerrit.cloudera.org/#/c/12365/

  was:
https://gerrit.cloudera.org/#/c/12549/

https://gerrit.cloudera.org/#/c/12365/


> Impala Doc: Doc automatic invalidates using metastore notification events
> -
>
> Key: IMPALA-7974
> URL: https://issues.apache.org/jira/browse/IMPALA-7974
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>  Labels: future_release_doc, in_32
>
> https://gerrit.cloudera.org/#/c/12549/
> https://gerrit.cloudera.org/#/c/12118/
> https://gerrit.cloudera.org/#/c/12365/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-7974) Impala Doc: Doc automatic invalidates using metastore notification events

2019-02-25 Thread Alex Rodoni (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni updated IMPALA-7974:

Description: 
https://gerrit.cloudera.org/#/c/12549/

https://gerrit.cloudera.org/#/c/12365/

  was:https://gerrit.cloudera.org/#/c/12549/


> Impala Doc: Doc automatic invalidates using metastore notification events
> -
>
> Key: IMPALA-7974
> URL: https://issues.apache.org/jira/browse/IMPALA-7974
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>  Labels: future_release_doc, in_32
>
> https://gerrit.cloudera.org/#/c/12549/
> https://gerrit.cloudera.org/#/c/12365/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-7974) Impala Doc: Doc automatic invalidates using metastore notification events

2019-02-25 Thread Alex Rodoni (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni updated IMPALA-7974:

Description: https://gerrit.cloudera.org/#/c/12549/

> Impala Doc: Doc automatic invalidates using metastore notification events
> -
>
> Key: IMPALA-7974
> URL: https://issues.apache.org/jira/browse/IMPALA-7974
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>  Labels: future_release_doc, in_32
>
> https://gerrit.cloudera.org/#/c/12549/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Created] (IMPALA-8246) Add a table with Hadoop components / versions and the type of timestamp they write in Parquet, and the problems we can expect when reading these with Impala.

2019-02-25 Thread Alex Rodoni (JIRA)
Alex Rodoni created IMPALA-8246:
---

 Summary: Add a table with Hadoop components / versions and the 
type of timestamp they write in Parquet, and the problems we can expect when 
reading these with Impala.
 Key: IMPALA-8246
 URL: https://issues.apache.org/jira/browse/IMPALA-8246
 Project: IMPALA
  Issue Type: Improvement
  Components: Docs
Reporter: Alex Rodoni
Assignee: Alex Rodoni






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (IMPALA-8246) Add a table for timestamp written to Parquet and the problems when read by Impala

2019-02-25 Thread Alex Rodoni (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni updated IMPALA-8246:

Summary: Add a table for timestamp written to Parquet and the problems when 
read by Impala  (was: Add a table with Hadoop components / versions and the 
type of timestamp they write in Parquet, and the problems we can expect when 
reading these with Impala.)

> Add a table for timestamp written to Parquet and the problems when read by 
> Impala
> -
>
> Key: IMPALA-8246
> URL: https://issues.apache.org/jira/browse/IMPALA-8246
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>
> Add a table with Hadoop components / versions and the type of timestamp they 
> write in Parquet, and the problems we can expect when reading these with 
> Impala.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-8246) Add a table for timestamp written to Parquet and the problems when read by Impala

2019-02-25 Thread Alex Rodoni (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777415#comment-16777415
 ] 

Alex Rodoni commented on IMPALA-8246:
-

[~csringhofer]


> Add a table for timestamp written to Parquet and the problems when read by 
> Impala
> -
>
> Key: IMPALA-8246
> URL: https://issues.apache.org/jira/browse/IMPALA-8246
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>
> Add a table with Hadoop components / versions and the type of timestamp they 
> write in Parquet, and the problems we can expect when reading these with 
> Impala.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Updated] (IMPALA-8246) Add a table with Hadoop components / versions and the type of timestamp they write in Parquet, and the problems we can expect when reading these with Impala.

2019-02-25 Thread Alex Rodoni (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni updated IMPALA-8246:

Description: Add a table with Hadoop components / versions and the type of 
timestamp they write in Parquet, and the problems we can expect when reading 
these with Impala.

> Add a table with Hadoop components / versions and the type of timestamp they 
> write in Parquet, and the problems we can expect when reading these with 
> Impala.
> -
>
> Key: IMPALA-8246
> URL: https://issues.apache.org/jira/browse/IMPALA-8246
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Docs
>Reporter: Alex Rodoni
>Assignee: Alex Rodoni
>Priority: Major
>
> Add a table with Hadoop components / versions and the type of timestamp they 
> write in Parquet, and the problems we can expect when reading these with 
> Impala.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Created] (IMPALA-8246) Add a table with Hadoop components / versions and the type of timestamp they write in Parquet, and the problems we can expect when reading these with Impala.

2019-02-25 Thread Alex Rodoni (JIRA)
Alex Rodoni created IMPALA-8246:
---

 Summary: Add a table with Hadoop components / versions and the 
type of timestamp they write in Parquet, and the problems we can expect when 
reading these with Impala.
 Key: IMPALA-8246
 URL: https://issues.apache.org/jira/browse/IMPALA-8246
 Project: IMPALA
  Issue Type: Improvement
  Components: Docs
Reporter: Alex Rodoni
Assignee: Alex Rodoni






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-6240) PARQUET_ARRAY_RESOLUTION query option not documented

2019-02-25 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777396#comment-16777396
 ] 

ASF subversion and git services commented on IMPALA-6240:
-

Commit 293578588e01a2cc9944d99ca2e42955f9003be8 in impala's branch 
refs/heads/2.x from Alex Rodoni
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2935785 ]

[DOCS] Corrected the supported values for parquet_array_resolution

Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356

Cherry-pick the following commit first, otherwise the doc doesn't exist.
I have change the description of the default value that
PARQUET_ARRAY_RESOLUTION defaults to TWO_LEVEL_THEN_THREE_LEVEL in 2.x.

IMPALA-6240: [DOCS] Document PARQUET_ARRAY_RESOLUTION query option

Cherry-picks: not for 2.x
Change-Id: I12696b609609ea16c05d8b7e84b2bae0be6d6cb5
Reviewed-on: http://gerrit.cloudera.org:8080/9534
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins

[DOCS] Corrected the supported values for parquet_array_resolution

Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356
Reviewed-on: http://gerrit.cloudera.org:8080/10840
Reviewed-by: Alex Rodoni 
Tested-by: Impala Public Jenkins 
Reviewed-on: http://gerrit.cloudera.org:8080/12571
Reviewed-by: Tim Armstrong 


> PARQUET_ARRAY_RESOLUTION query option not documented
> 
>
> Key: IMPALA-6240
> URL: https://issues.apache.org/jira/browse/IMPALA-6240
> Project: IMPALA
>  Issue Type: Documentation
>  Components: Docs
>Affects Versions: Impala 2.9.0, Impala 2.10.0
>Reporter: Petter von Dolwitz
>Assignee: Alex Rodoni
>Priority: Major
>  Labels: documentation
>
> The query option PARQUET_ARRAY_RESOLUTION that was introduced when resolving 
> IMPALA-4725 is not included in the documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Resolved] (IMPALA-8212) Crash during startup in kudu::security::CanonicalizeKrb5Principal()

2019-02-25 Thread Michael Ho (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Ho resolved IMPALA-8212.

   Resolution: Fixed
Fix Version/s: Impala 3.2.0

> Crash during startup in kudu::security::CanonicalizeKrb5Principal()
> ---
>
> Key: IMPALA-8212
> URL: https://issues.apache.org/jira/browse/IMPALA-8212
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.2.0
> Environment: CentOS Linux release 7.4.1708 (Core)
> Linux vc0512.halxg.cloudera.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 
> 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Blocker
>  Labels: crash
> Fix For: Impala 3.2.0
>
> Attachments: gdb-core-60055.txt, gdb.txt, hs_err_pid60055.log, 
> hs_err_pid65365.log, 
> impalad.vc0512.halxg.cloudera.com.impala.log.INFO.20190218-140034.65365, 
> impalad.vc0513.halxg.cloudera.com.impala.log.INFO.20190216-142536.60055
>
>
> I saw this crash twice will working on the stress test. It *seems* to happen 
> when the stress infrastructure switches the service to a debug build, 
> restarts the service, then starts running queries. I haven't seen it happen 
> once the service is up and running for a while.
> {noformat}
> #0  0x7fb03e1fa1f7 in raise () from sysroot/lib64/libc.so.6
> #1  0x7fb03e1fb8e8 in abort () from sysroot/lib64/libc.so.6
> #2  0x7fb041159185 in os::abort(bool) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #3  0x7fb0412fb593 in VMError::report_and_die() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #4  0x7fb04115e68f in JVM_handle_linux_signal () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #5  0x7fb041154be3 in signalHandler(int, siginfo*, void*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #6  
> #7  0x048d0a53 in 
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
>  unsigned long, int) ()
> #8  0x048d0aec in 
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned 
> long) ()
> #9  0x04a0b4c0 in tc_free ()
> #10 0x7fb040d32933 in ElfDecoder::demangle(char const*, char*, int) () 
> from sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #11 0x7fb040d3222a in Decoder::demangle(char const*, char*, int) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #12 0x7fb04115695d in os::dll_address_to_function_name(unsigned char*, 
> char*, int, int*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #13 0x7fb040dc0222 in frame::print_C_frame(outputStream*, char*, int, 
> unsigned char*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #14 0x7fb040d2e925 in print_native_stack(outputStream*, frame, Thread*, 
> char*, int) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #15 0x7fb0412f9cc8 in VMError::report(outputStream*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #16 0x7fb0412fb18a in VMError::report_and_die() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #17 0x7fb04115e68f in JVM_handle_linux_signal () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #18 0x7fb041154be3 in signalHandler(int, siginfo*, void*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #19 
> #20 0x048d0a53 in 
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
>  unsigned long, int) ()
> #21 0x048d0aec in 
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned 
> long) ()
> #22 0x04a0b4c0 in tc_free ()
> #23 0x7fb03e5915dd in pthread_attr_destroy () from 
> sysroot/lib64/libpthread.so.0
> #24 0x7fb04115e49f in current_stack_region(unsigned char**, unsigned 
> long*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #25 0x7fb04115e535 in os::current_stack_base() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #26 0x7fb0412faeb4 in VMError::report(outputStream*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #27 0x7fb0412fb18a in VMError::report_and_die() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #28 0x7fb04115e68f in JVM_handle_linux_signal () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> 

[jira] [Resolved] (IMPALA-8212) Crash during startup in kudu::security::CanonicalizeKrb5Principal()

2019-02-25 Thread Michael Ho (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Ho resolved IMPALA-8212.

   Resolution: Fixed
Fix Version/s: Impala 3.2.0

> Crash during startup in kudu::security::CanonicalizeKrb5Principal()
> ---
>
> Key: IMPALA-8212
> URL: https://issues.apache.org/jira/browse/IMPALA-8212
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.2.0
> Environment: CentOS Linux release 7.4.1708 (Core)
> Linux vc0512.halxg.cloudera.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 
> 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Blocker
>  Labels: crash
> Fix For: Impala 3.2.0
>
> Attachments: gdb-core-60055.txt, gdb.txt, hs_err_pid60055.log, 
> hs_err_pid65365.log, 
> impalad.vc0512.halxg.cloudera.com.impala.log.INFO.20190218-140034.65365, 
> impalad.vc0513.halxg.cloudera.com.impala.log.INFO.20190216-142536.60055
>
>
> I saw this crash twice will working on the stress test. It *seems* to happen 
> when the stress infrastructure switches the service to a debug build, 
> restarts the service, then starts running queries. I haven't seen it happen 
> once the service is up and running for a while.
> {noformat}
> #0  0x7fb03e1fa1f7 in raise () from sysroot/lib64/libc.so.6
> #1  0x7fb03e1fb8e8 in abort () from sysroot/lib64/libc.so.6
> #2  0x7fb041159185 in os::abort(bool) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #3  0x7fb0412fb593 in VMError::report_and_die() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #4  0x7fb04115e68f in JVM_handle_linux_signal () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #5  0x7fb041154be3 in signalHandler(int, siginfo*, void*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #6  
> #7  0x048d0a53 in 
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
>  unsigned long, int) ()
> #8  0x048d0aec in 
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned 
> long) ()
> #9  0x04a0b4c0 in tc_free ()
> #10 0x7fb040d32933 in ElfDecoder::demangle(char const*, char*, int) () 
> from sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #11 0x7fb040d3222a in Decoder::demangle(char const*, char*, int) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #12 0x7fb04115695d in os::dll_address_to_function_name(unsigned char*, 
> char*, int, int*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #13 0x7fb040dc0222 in frame::print_C_frame(outputStream*, char*, int, 
> unsigned char*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #14 0x7fb040d2e925 in print_native_stack(outputStream*, frame, Thread*, 
> char*, int) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #15 0x7fb0412f9cc8 in VMError::report(outputStream*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #16 0x7fb0412fb18a in VMError::report_and_die() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #17 0x7fb04115e68f in JVM_handle_linux_signal () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #18 0x7fb041154be3 in signalHandler(int, siginfo*, void*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #19 
> #20 0x048d0a53 in 
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
>  unsigned long, int) ()
> #21 0x048d0aec in 
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned 
> long) ()
> #22 0x04a0b4c0 in tc_free ()
> #23 0x7fb03e5915dd in pthread_attr_destroy () from 
> sysroot/lib64/libpthread.so.0
> #24 0x7fb04115e49f in current_stack_region(unsigned char**, unsigned 
> long*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #25 0x7fb04115e535 in os::current_stack_base() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #26 0x7fb0412faeb4 in VMError::report(outputStream*) () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #27 0x7fb0412fb18a in VMError::report_and_die() () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> #28 0x7fb04115e68f in JVM_handle_linux_signal () from 
> sysroot/usr/java/jdk1.8.0_141-cloudera/jre/lib/amd64/server/libjvm.so
> 

[jira] [Updated] (IMPALA-8242) Support Iceberg on S3

2019-02-25 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong updated IMPALA-8242:
--
Description: http://iceberg.incubator.apache.org/

> Support Iceberg on S3
> -
>
> Key: IMPALA-8242
> URL: https://issues.apache.org/jira/browse/IMPALA-8242
> Project: IMPALA
>  Issue Type: New Feature
>Reporter: Quanlong Huang
>Priority: Major
>
> http://iceberg.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Created] (IMPALA-8245) Add hostname to timeout error message in HdfsMonitoredOps

2019-02-25 Thread Joe McDonnell (JIRA)
Joe McDonnell created IMPALA-8245:
-

 Summary: Add hostname to timeout error message in HdfsMonitoredOps
 Key: IMPALA-8245
 URL: https://issues.apache.org/jira/browse/IMPALA-8245
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Joe McDonnell


If a DiskIo operation times out, it generates a 
TErrorCode::THREAD_POOL_TASK_TIMED_OUT or TErrorCode::THREAD_POOL_SUBMIT_FAILED 
error codes. These call GetDescription() to get DiskIo related context. That 
information should include the hostname where the error occurred to allow 
tracking down a problematic host that is seeing DiskIo timeouts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Assigned] (IMPALA-8245) Add hostname to timeout error message in HdfsMonitoredOps

2019-02-25 Thread Pooja Nilangekar (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pooja Nilangekar reassigned IMPALA-8245:


Assignee: Pooja Nilangekar

> Add hostname to timeout error message in HdfsMonitoredOps
> -
>
> Key: IMPALA-8245
> URL: https://issues.apache.org/jira/browse/IMPALA-8245
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Pooja Nilangekar
>Priority: Major
>
> If a DiskIo operation times out, it generates a 
> TErrorCode::THREAD_POOL_TASK_TIMED_OUT or 
> TErrorCode::THREAD_POOL_SUBMIT_FAILED error codes. These call 
> GetDescription() to get DiskIo related context. That information should 
> include the hostname where the error occurred to allow tracking down a 
> problematic host that is seeing DiskIo timeouts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Created] (IMPALA-8245) Add hostname to timeout error message in HdfsMonitoredOps

2019-02-25 Thread Joe McDonnell (JIRA)
Joe McDonnell created IMPALA-8245:
-

 Summary: Add hostname to timeout error message in HdfsMonitoredOps
 Key: IMPALA-8245
 URL: https://issues.apache.org/jira/browse/IMPALA-8245
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Joe McDonnell


If a DiskIo operation times out, it generates a 
TErrorCode::THREAD_POOL_TASK_TIMED_OUT or TErrorCode::THREAD_POOL_SUBMIT_FAILED 
error codes. These call GetDescription() to get DiskIo related context. That 
information should include the hostname where the error occurred to allow 
tracking down a problematic host that is seeing DiskIo timeouts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IMPALA-8241) from_utc_timestamp returns inconsistent results with Hive

2019-02-25 Thread Tim Armstrong (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777242#comment-16777242
 ] 

Tim Armstrong commented on IMPALA-8241:
---

I think this is the thing we have --use_local_tz_for_unix_timestamp_conversions 
for

If I run impala with "start-impala-cluster.py 
--impalad_args=--use_local_tz_for_unix_timestamp_conversions=true", i get:
{noformat}
[localhost:21000] default> select from_utc_timestamp(cast(40 * 3600.0 as 
timestamp), 'EST');
Query: select from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'EST')
Query submitted at: 2019-02-25 11:44:01 (Coordinator: 
http://tarmstrong-box:25000)
Query progress can be monitored at: 
http://tarmstrong-box:25000/query_plan?query_id=9e435928e630b26d:2489fb78
+---+
| from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'est') |
+---+
| 2015-08-19 04:00:00   |
+---+
Fetched 1 row(s) in 0.30s

{noformat}

[~attilaj] can confirm maybe

> from_utc_timestamp returns inconsistent results with Hive
> -
>
> Key: IMPALA-8241
> URL: https://issues.apache.org/jira/browse/IMPALA-8241
> Project: IMPALA
>  Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Major
>
> This can be reproduced in both master and 2.x branches.
> {code}
> [localhost:21000] default> select from_utc_timestamp(cast(40 * 3600.0 as 
> timestamp), 'EST');
> Query: select from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'EST')
> Query submitted at: 2019-02-23 17:27:02 (Coordinator: 
> http://impala-jenkins-slave-02:25000)
> Query progress can be monitored at: 
> http://impala-jenkins-slave-02:25000/query_plan?query_id=f476c87a904f281:71588a24
> +---+
> | from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'est') |
> +---+
> | 2015-08-19 11:00:00   |
> +---+
> Fetched 1 row(s) in 0.64s
> {code}
> {code}
> hive> select from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'EST');
> OK
> 2015-08-19 04:00:00
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Resolved] (IMPALA-8064) test_min_max_filters is flaky

2019-02-25 Thread Pooja Nilangekar (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pooja Nilangekar resolved IMPALA-8064.
--
Resolution: Fixed

> test_min_max_filters is flaky 
> --
>
> Key: IMPALA-8064
> URL: https://issues.apache.org/jira/browse/IMPALA-8064
> Project: IMPALA
>  Issue Type: Bug
>Reporter: Pooja Nilangekar
>Assignee: Pooja Nilangekar
>Priority: Blocker
>  Labels: broken-build, flaky-test
> Fix For: Impala 3.2.0
>
> Attachments: profile.txt
>
>
> The following configuration of the test_min_max_filters:
> {code:java}
> query_test.test_runtime_filters.TestMinMaxFilters.test_min_max_filters[protocol:
>  beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 
> 0} | table_format: kudu/none]{code}
> It produces a higher aggregation of sum over the proberows than expected:
> {code:java}
> query_test/test_runtime_filters.py:113: in test_min_max_filters 
> self.run_test_case('QueryTest/min_max_filters', vector) 
> common/impala_test_suite.py:518: in run_test_case 
> update_section=pytest.config.option.update_results) 
> common/test_result_verifier.py:612: in verify_runtime_profile % 
> (function, field, expected_value, actual_value, actual)) 
> E   AssertionError: Aggregation of SUM over ProbeRows did not match expected 
> results. 
> E   EXPECTED VALUE: E   619 
> EACTUAL VALUE: E   652
> {code}
> This test was introduced in the patch for IMPALA-6533. The failure occurred 
> during an ASAN build. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Resolved] (IMPALA-8064) test_min_max_filters is flaky

2019-02-25 Thread Pooja Nilangekar (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pooja Nilangekar resolved IMPALA-8064.
--
Resolution: Fixed

> test_min_max_filters is flaky 
> --
>
> Key: IMPALA-8064
> URL: https://issues.apache.org/jira/browse/IMPALA-8064
> Project: IMPALA
>  Issue Type: Bug
>Reporter: Pooja Nilangekar
>Assignee: Pooja Nilangekar
>Priority: Blocker
>  Labels: broken-build, flaky-test
> Fix For: Impala 3.2.0
>
> Attachments: profile.txt
>
>
> The following configuration of the test_min_max_filters:
> {code:java}
> query_test.test_runtime_filters.TestMinMaxFilters.test_min_max_filters[protocol:
>  beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 
> 0} | table_format: kudu/none]{code}
> It produces a higher aggregation of sum over the proberows than expected:
> {code:java}
> query_test/test_runtime_filters.py:113: in test_min_max_filters 
> self.run_test_case('QueryTest/min_max_filters', vector) 
> common/impala_test_suite.py:518: in run_test_case 
> update_section=pytest.config.option.update_results) 
> common/test_result_verifier.py:612: in verify_runtime_profile % 
> (function, field, expected_value, actual_value, actual)) 
> E   AssertionError: Aggregation of SUM over ProbeRows did not match expected 
> results. 
> E   EXPECTED VALUE: E   619 
> EACTUAL VALUE: E   652
> {code}
> This test was introduced in the patch for IMPALA-6533. The failure occurred 
> during an ASAN build. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (IMPALA-8244) Toolchain build fails to publish binaries even if asked to do so

2019-02-25 Thread Laszlo Gaal (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-8244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Gaal reassigned IMPALA-8244:
---

Assignee: Laszlo Gaal

> Toolchain build fails to publish binaries even if asked to do so
> 
>
> Key: IMPALA-8244
> URL: https://issues.apache.org/jira/browse/IMPALA-8244
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Laszlo Gaal
>Assignee: Laszlo Gaal
>Priority: Critical
>  Labels: native-toolchain, toolchain
>
> The recent commit to native-toolchain that enabled the publishing of binary 
> artifacts from Docker-based toolchain builds 
> ([https://github.com/cloudera/native-toolchain/commit/eeccbcab224fca5c25a7b78898444037b94d88b7)]
>  changed the logic that gates the publishing calls.
> Internal builds on Jenkins (using the same logic as before the change) now 
> don't even attempt publishing even if parameterized to do so. Excerpt from 
> the build log:
> {code:java}
> 02:19:44.605 install -d 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.608 install -m 644 -p libcctz.a 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.860 install -d 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.863 install -m 644 -p libcctz.so.2.2 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.880 make: Leaving directory 
> `/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz/cctz-2.2/build'
> 02:19:45.399 Cleaning 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
>  ...
> 02:19:45.400 Entering directory 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
> 02:19:45.457 Removing cctz-2.2.tar.gz
> 02:19:45.457 Removing cctz-2.2/
> 02:19:45.459 Returning to directory 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain
> 02:19:45.459 # Build Complete cctz-2.2
> 02:19:45.459 
> ###
> 02:19:45.549 + popd
> 02:19:45.549 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem
> 02:19:45.579 Archiving artifacts
> 02:19:45.781 [description-setter] Description set: 
> ***redacted**
> 02:19:46.010 [WS-CLEANUP] Deleting project workspace...
> 02:19:46.010 [WS-CLEANUP] Deferred wipeout is used...
> 02:19:46.057 [WS-CLEANUP] done
> 02:19:46.057 Email was triggered for: Success
> 02:19:46.057 Sending email for trigger: Success
> 02:19:46.523 Sending email to: laszlo.g...@cloudera.com
> 02:19:46.551 Finished: SUCCESS{code}
> The happy case with the missing steps (copied from another job that published 
> the bits successfully):
> {code:java}
> 02:26:22.083 install -d 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:26:22.085 install -m 644 -p libcctz.a 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:26:22.338 install -d 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:26:22.340 install -m 644 -p libcctz.so.2.2 
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:26:22.358 make: Leaving directory 
> `/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz/cctz-2.2/build'
> 02:26:24.297 [INFO] Scanning for projects...
> 02:26:24.545 [INFO]   
>   
> 02:26:24.545 [INFO] 
> 
> 02:26:24.545 [INFO] Building Maven Stub Project (No POM) 1
> 02:26:24.546 [INFO] 
> 
> 02:26:24.552 [INFO] 
> 02:26:24.552 [INFO] --- maven-deploy-plugin:2.7:deploy-file (default-cli) @ 
> standalone-pom ---
> 02:26:25.079 Uploading: 
> 

[jira] [Created] (IMPALA-8244) Toolchain build fails to publish binaries even if asked to do so

2019-02-25 Thread Laszlo Gaal (JIRA)
Laszlo Gaal created IMPALA-8244:
---

 Summary: Toolchain build fails to publish binaries even if asked 
to do so
 Key: IMPALA-8244
 URL: https://issues.apache.org/jira/browse/IMPALA-8244
 Project: IMPALA
  Issue Type: Bug
  Components: Infrastructure
Affects Versions: Impala 3.2.0
Reporter: Laszlo Gaal


The recent commit to native-toolchain that enabled the publishing of binary 
artifacts from Docker-based toolchain builds 
([https://github.com/cloudera/native-toolchain/commit/eeccbcab224fca5c25a7b78898444037b94d88b7)]
 changed the logic that gates the publishing calls.

Internal builds on Jenkins (using the same logic as before the change) now 
don't even attempt publishing even if parameterized to do so. Excerpt from the 
build log:
{code:java}
02:19:44.605 install -d 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:19:44.608 install -m 644 -p libcctz.a 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:19:44.860 install -d 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:19:44.863 install -m 644 -p libcctz.so.2.2 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:19:44.880 make: Leaving directory 
`/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz/cctz-2.2/build'
02:19:45.399 Cleaning 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
 ...
02:19:45.400 Entering directory 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
02:19:45.457 Removing cctz-2.2.tar.gz
02:19:45.457 Removing cctz-2.2/
02:19:45.459 Returning to directory 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain
02:19:45.459 # Build Complete cctz-2.2
02:19:45.459 
###
02:19:45.549 + popd
02:19:45.549 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem
02:19:45.579 Archiving artifacts
02:19:45.781 [description-setter] Description set: 
***redacted**
02:19:46.010 [WS-CLEANUP] Deleting project workspace...
02:19:46.010 [WS-CLEANUP] Deferred wipeout is used...
02:19:46.057 [WS-CLEANUP] done
02:19:46.057 Email was triggered for: Success
02:19:46.057 Sending email for trigger: Success
02:19:46.523 Sending email to: laszlo.g...@cloudera.com
02:19:46.551 Finished: SUCCESS{code}
The happy case with the missing steps (copied from another job that published 
the bits successfully):
{code:java}
02:26:22.083 install -d 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:26:22.085 install -m 644 -p libcctz.a 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:26:22.338 install -d 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:26:22.340 install -m 644 -p libcctz.so.2.2 
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
02:26:22.358 make: Leaving directory 
`/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz/cctz-2.2/build'
02:26:24.297 [INFO] Scanning for projects...
02:26:24.545 [INFO] 

02:26:24.545 [INFO] 

02:26:24.545 [INFO] Building Maven Stub Project (No POM) 1
02:26:24.546 [INFO] 

02:26:24.552 [INFO] 
02:26:24.552 [INFO] --- maven-deploy-plugin:2.7:deploy-file (default-cli) @ 
standalone-pom ---
02:26:25.079 Uploading: 
http://maven.redacted/toolchain/cctz/2.2-gcc-4.9.2-253-1ffcf70e3f/cctz-2.2-gcc-4.9.2-253-1ffcf70e3f-ec2-package-centos-5.tar.gz
02:26:25.400 Uploaded: 
http://maven.redacted/toolchain/cctz/2.2-gcc-4.9.2-253-1ffcf70e3f/cctz-2.2-gcc-4.9.2-253-1ffcf70e3f-ec2-package-centos-5.tar.gz
 (2681 KB at 8325.9 KB/sec)
02:26:25.401 Uploading: 
http://maven.redacted/toolchain/cctz/2.2-gcc-4.9.2-253-1ffcf70e3f/cctz-2.2-gcc-4.9.2-253-1ffcf70e3f.pom
02:26:25.421 Uploaded: