[GitHub] incubator-hawq-site pull request #16: HAWQ-1638. Correct typo and use full n...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-hawq-site/pull/16 ---
[jira] [Closed] (HAWQ-1627) Support setting the max protocol message size when talking with HDFS
[
https://issues.apache.org/jira/browse/HAWQ-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1627.
---
Resolution: Fixed
fixed
> Support setting the max protocol message size when talking with HDFS
>
>
> Key: HAWQ-1627
> URL: https://issues.apache.org/jira/browse/HAWQ-1627
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: libhdfs
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> Now, the max size of protocol message in libhdfs is 64MB, and it cannot be
> adjusted.
> When the max size is reached (accessing a very big HDFS table/file), will see
> the following lines in hawq master log:
> {code}
> 2018-06-20 11:21:56.768003
> CST,,,p75703,th-8481004160,,,seg-1,"LOG","0","3rd party error
> log:
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:208] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to
> disable these warnings), see CodedInputStream::SetTotalBytesLimit() in
> google/protobuf/io/coded_stream.h.""SysLoggerMain","syslogger.c",518,
> 2018-06-20 11:21:56.771657
> CST,,,p75703,th-8481004160,,,seg-1,"LOG","0","3rd party error
> log:
> 2018-06-20 11:21:56.771492, p75751, th0x7fffcd7303c0, ERROR Failed to invoke
> RPC call ""getFsStats"" on server ""localhost:9000"":
> RpcChannel.cpp: 783: HdfsRpcException: RPC channel to ""localhost:9000"" got
> protocol mismatch: RPC channel cannot parse response header.
> @ Hdfs::Internal::RpcChannelImpl::readOneResponse(bool)
> @ Hdfs::Internal::RpcChannelImpl::checkOneResponse()
> @
> Hdfs::Internal::RpcChannelImpl::invokeInternal(std::__1::shared_ptr)
> @
> Hdfs::Internal::RpcChannelImpl:""SysLoggerMain","syslogger.c",518,
> 2018-06-20 11:21:56.771711
> CST,,,p75703,th-8481004160,,,seg-1,"LOG","0","3rd party error
> log:
> :invoke(Hdfs::Internal::RpcCall const&)
> @ Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&)
> @ Hdfs::Internal::NamenodeImpl::getFsStats()
> @ Hdfs::Internal::NamenodeProxy::getFsStats()
> @ Hdfs::Internal::FileSystemImpl::getFsStats()
> @ Hdfs::Internal::FileSystemImpl::connect()
> @ Hdfs::FileSystem::connect(char const*, char const*, char const*)
> @ Hdfs::FileSystem::connect(char const*)
> @ hdfsBuilderConnect
> @ gpfs_hdfs_connect
> @ HdfsConnect
> @ HdfsGetConnection
> @ HdfsGetFileBlockLocati""SysLoggerMain","syslogger.c",518,
> {code}
> Considering HDFS has a guc *"ipc.maximum.data.length"* to set it, HAWQ should
> also add a guc for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HAWQ-1638) Issues with website
[ https://issues.apache.org/jira/browse/HAWQ-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537839#comment-16537839 ] ASF GitHub Bot commented on HAWQ-1638: -- Github user changleicn commented on the issue: https://github.com/apache/incubator-hawq-site/pull/16 looks good! > Issues with website > --- > > Key: HAWQ-1638 > URL: https://issues.apache.org/jira/browse/HAWQ-1638 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Sebb >Assignee: Radar Lei >Priority: Major > > The HAWQ page looks nice, however there are a few problems with it. > The phrase > "Plus, HAWQ® works Apache MADlib) machine learning libraries" > does not read well. Something missing? > The first/main references to ASF products such as Hadoop, YARN etc must use > the full name, i.e. Apache Hadoop etc. > The download section does not have any link to the KEYS file, nor any > instructions on how to use the KEYS+sig or hashes to validate downloads. > The download section still includes references to MD5 hashes. > These are deprecated and can be removed for older releases that have other > hashes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq-site issue #16: HAWQ-1638. Correct typo and use full name for...
Github user changleicn commented on the issue: https://github.com/apache/incubator-hawq-site/pull/16 looks good! ---
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user lavjain commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r201091933
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/UGICache.java ---
@@ -0,0 +1,129 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.DelayQueue;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.security.UserGroupInformation;
+
+public class UGICache {
+
+private static final Log LOG = LogFactory.getLog(UGICache.class);
+private static Map cache = new
ConcurrentHashMap<>();
+//private static DelayQueue delayQueue = new
DelayQueue<>();
+private static DelayQueue[] delayQueues = new
DelayQueue<>[64];
+public static long UGI_CACHE_EXPIRY = 15 * 1 * 1000L; // 15 Minutes
+
+public UGICache() {
+for (int i = 0; i < delayQueues.length; i++) {
+delayQueues[i] = new DelayQueue<>();
--- End diff --
w\ @benchristel Will change `DelayQueues` to be not static.
---
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r201090455
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/SegmentTransactionId.java
---
@@ -0,0 +1,60 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+public class SegmentTransactionId {
+
+private Integer segmentId;
+private String transactionId;
--- End diff --
We might need it in the future. It doesn't hurt to leave in there for now.
---
[jira] [Commented] (HAWQ-1638) Issues with website
[ https://issues.apache.org/jira/browse/HAWQ-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536746#comment-16536746 ] ASF GitHub Bot commented on HAWQ-1638: -- Github user radarwave commented on the issue: https://github.com/apache/incubator-hawq-site/pull/16 Download links fix will be update later. @changleicn @edespino @jiny2 Would you help to review? Thanks. > Issues with website > --- > > Key: HAWQ-1638 > URL: https://issues.apache.org/jira/browse/HAWQ-1638 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Sebb >Assignee: Radar Lei >Priority: Major > > The HAWQ page looks nice, however there are a few problems with it. > The phrase > "Plus, HAWQ® works Apache MADlib) machine learning libraries" > does not read well. Something missing? > The first/main references to ASF products such as Hadoop, YARN etc must use > the full name, i.e. Apache Hadoop etc. > The download section does not have any link to the KEYS file, nor any > instructions on how to use the KEYS+sig or hashes to validate downloads. > The download section still includes references to MD5 hashes. > These are deprecated and can be removed for older releases that have other > hashes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq-site issue #16: HAWQ-1638. Correct typo and use full name for...
Github user radarwave commented on the issue: https://github.com/apache/incubator-hawq-site/pull/16 Download links fix will be update later. @changleicn @edespino @jiny2 Would you help to review? Thanks. ---
[GitHub] incubator-hawq-site pull request #16: HAWQ-1638. Correct typo and use full n...
GitHub user radarwave opened a pull request: https://github.com/apache/incubator-hawq-site/pull/16 HAWQ-1638. Correct typo and use full name for ASF products Correct typo for madlib and use full name for ASF products with the first/main references. You can merge this pull request into a Git repository by running: $ git pull https://github.com/radarwave/incubator-hawq-site HAWQ-1638 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-site/pull/16.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16 commit c6a20e535d9a36d39977269b00b2e6cea05687f8 Author: rlei Date: 2018-07-09T09:52:04Z HAWQ-1638. Correct typo and use full name for ASF products with the first/main references. ---
[jira] [Commented] (HAWQ-1638) Issues with website
[ https://issues.apache.org/jira/browse/HAWQ-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536742#comment-16536742 ] ASF GitHub Bot commented on HAWQ-1638: -- GitHub user radarwave opened a pull request: https://github.com/apache/incubator-hawq-site/pull/16 HAWQ-1638. Correct typo and use full name for ASF products Correct typo for madlib and use full name for ASF products with the first/main references. You can merge this pull request into a Git repository by running: $ git pull https://github.com/radarwave/incubator-hawq-site HAWQ-1638 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-site/pull/16.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16 commit c6a20e535d9a36d39977269b00b2e6cea05687f8 Author: rlei Date: 2018-07-09T09:52:04Z HAWQ-1638. Correct typo and use full name for ASF products with the first/main references. > Issues with website > --- > > Key: HAWQ-1638 > URL: https://issues.apache.org/jira/browse/HAWQ-1638 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Sebb >Assignee: Radar Lei >Priority: Major > > The HAWQ page looks nice, however there are a few problems with it. > The phrase > "Plus, HAWQ® works Apache MADlib) machine learning libraries" > does not read well. Something missing? > The first/main references to ASF products such as Hadoop, YARN etc must use > the full name, i.e. Apache Hadoop etc. > The download section does not have any link to the KEYS file, nor any > instructions on how to use the KEYS+sig or hashes to validate downloads. > The download section still includes references to MD5 hashes. > These are deprecated and can be removed for older releases that have other > hashes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
