[
https://issues.apache.org/jira/browse/HAWQ-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215156#comment-15215156
]
ASF GitHub Bot commented on HAWQ-577:
-------------------------------------
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/522#discussion_r57657152
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/MetadataResponse.java
---
@@ -0,0 +1,94 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.List;
+
+import javax.ws.rs.WebApplicationException;
+import javax.ws.rs.core.StreamingOutput;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.Metadata;
+
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.map.annotate.JsonSerialize.Inclusion;
+
+
+/**
+ * Class for serializing metadata in JSON format. The class implements
+ * {@link StreamingOutput} so the serialization will be done in a stream
and not
+ * in one bulk, this in order to avoid running out of memory when
processing a
+ * lot of items.
+ */
+public class MetadataResponse implements StreamingOutput {
+
+ private static final Log Log =
LogFactory.getLog(MetadataResponse.class);
+ private static final String METADATA_DEFAULT_RESPONSE =
"{\"PXFMetadata\":[]}";
+
+ private List<Metadata> metadataList;
+
+ /**
+ * Constructs metadata response out of a metadata list
+ *
+ * @param metadataList metadata list
+ */
+ public MetadataResponse(List<Metadata> metadataList) {
+ this.metadataList = metadataList;
+ }
+
+ /**
+ * Serializes the metadata list in JSON, To be used as the result
string for HAWQ.
+ */
+ @Override
+ public void write(OutputStream output) throws IOException,
+ WebApplicationException {
+ DataOutputStream dos = new DataOutputStream(output);
+ ObjectMapper mapper = new ObjectMapper();
+ mapper.setSerializationInclusion(Inclusion.NON_EMPTY); // ignore
empty fields
+
+ if(metadataList == null || metadataList.isEmpty()) {
+ dos.write(METADATA_DEFAULT_RESPONSE.getBytes());
+ return;
+ }
+
+ dos.write("{\"PXFMetadata\":[".getBytes());
--- End diff --
I didn't want to do that. The default response can possibly change tomorrow
and wanted to decouple that from a valid response string.
> Stream PXF metadata response
> -----------------------------
>
> Key: HAWQ-577
> URL: https://issues.apache.org/jira/browse/HAWQ-577
> Project: Apache HAWQ
> Issue Type: Bug
> Components: PXF
> Reporter: Shivram Mani
> Assignee: Shivram Mani
>
> getMetadata api returns the metadata corresponding to the user specified
> pattern. There is no limit to the #of tables the pattern can correspond do
> and the current approach of building in memory the json object might not
> scale.
> We needed to serialize them inside a streaming object similar to the approach
> used for streaming the FragmentsResponse
> The same applies also for the debug function that prints metadata of all the
> items - if there are too many of them the StringBuilder will run out of
> memory. The solution in the fragments case was to print a log of one fragment
> at a time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
