When you say it does not work what error do you get ? do you see the rewrite rules for header getting triggered in the debug logs ? Also, what is the exception you get on the elasticsearch side.
Best, Sandeep On Fri, Oct 5, 2018 at 11:23 AM rabii lamriq <lam...@gmail.com> wrote: > Hi All, > > I tried to change the content-type of the http header in the > ElasticsearchDispatcher java class, but doesn't work :( > > Anyone can give help where we should change the content-type for make > "application/x-ndjson" in class of ElasticsearchDispatcher (attached) > > Regards > Rabii > > Le mar. 25 sept. 2018 à 16:20, rabii lamriq <lam...@gmail.com> a écrit : > >> I am not understand how to apply this for content-type. >> >> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <moresand...@gmail.com> a >> écrit : >> >>> This is an example in Webhdfs service def. >>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64 >>> >>> >>> >>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <lam...@gmail.com> wrote: >>> >>>> Hi Sandeep, >>>> >>>> Yes, I know that is possible, but I don't know how to rewrite the rule >>>> to access to the header and change the Content-type, I can't find any thing >>>> related to this customization. >>>> >>>> Regards >>>> Rabii >>>> >>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <moresand...@gmail.com> a >>>> écrit : >>>> >>>>> Yes, you can use the rewrite rules to update Headers, I don't remember >>>>> exactly which service uses it but there should be few services that >>>>> rewrite >>>>> headers. >>>>> >>>>> Best, >>>>> Sandeep >>>>> >>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <lam...@gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Rewrite rules for knox (link >>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>) >>>>>> can help me to change header? is it possible? >>>>>> >>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <lam...@gmail.com> a >>>>>> écrit : >>>>>> >>>>>>> Hello >>>>>>> >>>>>>> Any help please?? >>>>>>> >>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <lam...@gmail.com> a >>>>>>> écrit : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> thanks every One, for your feedbacks, >>>>>>>> >>>>>>>> Yes, ES accept both application/json and application/x-ndjson, >>>>>>>> there is no problem in ES ni in Grafana, when i test my query with >>>>>>>> curl to >>>>>>>> send POST to ES, it work well ether json or x-ndjson. >>>>>>>> >>>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't >>>>>>>> know why. so, since Grafana send their POST with only json and not >>>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search >>>>>>>> doesn't >>>>>>>> work. >>>>>>>> >>>>>>>> I think we have two solution: >>>>>>>> >>>>>>>> 1- make some thing to force Grafana change header of POST and put >>>>>>>> x-ndjson instead of json, or >>>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES. >>>>>>>> >>>>>>>> If some one have any idea of how to make this or other solution. >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> Rabii >>>>>>>> >>>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kris...@apache.org> a >>>>>>>> écrit : >>>>>>>> >>>>>>>>> https://github.com/elastic/elasticsearch/issues/25718 >>>>>>>>> >>>>>>>>> Elasticsearch endpoints should take both application/json and >>>>>>>>> application/x-ndjson. There might be something else going on here. >>>>>>>>> That is >>>>>>>>> why I asked for the Elasticsearch logs. >>>>>>>>> >>>>>>>>> It might not be necessary to rewrite the content type. >>>>>>>>> >>>>>>>>> Kevin Risden >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <lam...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> My problem is how to change the content-type=application/json to >>>>>>>>>> application/x-ndjson by rewriting the header in the knox. >>>>>>>>>> >>>>>>>>>> Because grafana send their query with content-type=application/json >>>>>>>>>> and knox return status 400, >>>>>>>>>> >>>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed >>>>>>>>>> that if i replace json by x-ndjson through knox it wirk well. >>>>>>>>>> >>>>>>>>>> Sence i can't change the content-type in grafana, is this >>>>>>>>>> possible by rewriting the header in knox to replace json by x-ndjson. >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Rabii >>>>>>>>>> >>>>>>>>>> Le ven. 21 sept. 2018 à 18:05, >>>>>>>>>> Kevin Risden >>>>>>>>>> <kris...@apache.org> a écrit : >>>>>>>>>> >>>>>>>>>>> Do you have the error from Elasticsearch logs? It might explain >>>>>>>>>>> if Knox or Grafana is sending a bad request. >>>>>>>>>>> >>>>>>>>>>> Kevin Risden >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré < >>>>>>>>>>> moresand...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I don't see any issues from Knox side, what error do you see in >>>>>>>>>>>> Grafana ? I think this could be because Grafana is expecting >>>>>>>>>>>> certain >>>>>>>>>>>> content-type. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <lam...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Sandeep >>>>>>>>>>>>> >>>>>>>>>>>>> thanks for your feedback, below is the deboug log for this curl >>>>>>>>>>>>> >>>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway >>>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: >>>>>>>>>>>>> https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch, >>>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http:// >>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch >>>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway >>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - >>>>>>>>>>>>> Dispatch request: >>>>>>>>>>>>> POST http:// >>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user> >>>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway >>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - >>>>>>>>>>>>> Dispatch response >>>>>>>>>>>>> status: 400 >>>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway >>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using >>>>>>>>>>>>> explicit >>>>>>>>>>>>> character set UTF-8 for entity of type application/json >>>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway >>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - >>>>>>>>>>>>> Inbound >>>>>>>>>>>>> response entity content type: application/json; charset=UTF-8 >>>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway >>>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST >>>>>>>>>>>>> /es5/<index_name>/_msearch >>>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway >>>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> Rabii >>>>>>>>>>>>> >>>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré < >>>>>>>>>>>>> moresand...@gmail.com> a écrit : >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello Rab, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will >>>>>>>>>>>>>> tell us exactly what is happening. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Sandeep >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq < >>>>>>>>>>>>>> lam...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch >>>>>>>>>>>>>>> using the bulk API with the newline delimited JSON (NDJSON) >>>>>>>>>>>>>>> format. >>>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html >>>>>>>>>>>>>>> like this: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST >>>>>>>>>>>>>>> "http://host_ES:900/index/_msearch"; --data-binary ’ >>>>>>>>>>>>>>> { first json doc} >>>>>>>>>>>>>>> {second json doc} >>>>>>>>>>>>>>> ’ >>>>>>>>>>>>>>> when bypassing Knox, The query work well either we made json >>>>>>>>>>>>>>> or x-ndjson, >>>>>>>>>>>>>>> However, through Knox gateway, if the >>>>>>>>>>>>>>> content-type=application/json, the query retun this error of >>>>>>>>>>>>>>> 400: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> HTTP/1.1 400 Bad Request >>>>>>>>>>>>>>> content-type: application/json >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the >>>>>>>>>>>>>>> bulk API, it work well. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My problem is that Grafana send their query with the >>>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is >>>>>>>>>>>>>>> possible to >>>>>>>>>>>>>>> make any thing on Knox to solve this problem? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>> Rab >>>>>>>>>>>>>>> >>>>>>>>>>>>>>
