Hi Andy, both .gpx and .kml files are written in xml, so you could open them and read it with notepad++, or with a simple python script, and use some regex or something to trim all of the coordinates to fewer decimal places. That would be a good way to incorporate this into your workflow...
On Sun, May 1, 2016 at 4:38 PM, Adam Lodge <[email protected]> wrote: > Andy, > > Happy to help, but unfortunately, we don’t have a solution in our hip > pocket. We have only encountered this kind of issue when transferring > larger numbers of records from legacy systems into Arches. Therefore, our > solution is a bit one-off. And, I am a bit surprised that geometry (that I > assume was) collected with GPS units would be voluminous enough to surpass > that threshold. > > That said, what I have done in the past is run the data through an > appropriate FME transformer or use Postgis (st_snaptogrid function) to > process the data in a manner that resolves the issue. > > Starting to move beyond my domain of expertise, but it seems likely that > there is a gdal function that you could bind to your python ETL/integration > code that may do the trick. Wish I could point you to something more > specific. Maybe someone else can? > > One other thing - I highly recommend you start with reducing precision > rather than generalizing. You preserve much more integrity of the data and > still likely achieve your goal. > > Adam > > -- > Adam Lodge > Geospatial Systems Consultant > Farallon Geographics > > On Sunday, May 1, 2016 at 3:16 PM, Andy Graham wrote: > > Thanks for the info Adam. I figured it was something like that. In this > case generalizing the gpx probably is acceptable. Do you guys know of or > have a quick and easy program/code to do this? I can manage this with a > few steps but I visualize this being part of the daily work flow so would > prefer more of a single step, "push a button and it's generalized" type > setup. Let me know. Thanks. > > Andy > On May 1, 2016 2:17 PM, "Adam Lodge" <[email protected]> wrote: > > Andy, > > There is a known limitation - Im guessing from your error message that > 32766 - to the number of characters that Elastic Search can process in a > single value. This translates into a geometry limitation because (I think) > ES indexes geometries as geojson. In short, I think it’s less about the > size of the file and more about the number of characters necessary to > describe the geometry. > > We have done some tricks to push geometries like these through. They > include: > - Reduce the precision of the captured geometries (thereby reducing the > number of characters after the decimal that gets stored for each vertex) > - Generalize the geometry (if that is acceptable.) That reduces the number > of vertices in the geometry. > > If you are able to send the file, I would be happy to take a look at it. > > Adam > > -- > Adam Lodge > Geospatial Systems Consultant > Farallon Geographics > 415.317.6625 > > On Sunday, May 1, 2016 at 2:08 PM, Andy Graham wrote: > > So I have been trying to add geometry to one of our resources by dragging > and dropping a .gpx file onto the map in the edit form for the resource. > > I can add it to the map just fine but when I go to save I eventually get > an error on the webpage that states: > > TransportError at > /resources/SURVEY.E1/default/1510042c-6888-4fc9-8658-cd5cee6a8bdb > > TransportError(500, u'IllegalArgumentException[Document contains at least one > immense term in field="geometries.label" (whose UTF8 encoding is longer than > the max length 32766), all of which were skipped. Please correct the > analyzer to not produce such terms. The prefix of the first immense term is: > \'[77, 85, 76, 84, 73, 76, 73, 78, 69, 83, 84, 82, 73, 78, 71, 40, 40, 45, > 49, 50, 48, 46, 53, 54, 57, 57, 53, 49, 48, 51]...\', original message: bytes > can be at most 32766 in length; got 100421]; nested: > MaxBytesLengthExceededException[bytes can be at most 32766 in length; got > 100421]; ') > > > From what I can tell it appears as though there is a size limit on the > files used to create the geometry? I have done this successfully with > smaller gpx files, this current one is about 300KB. I have converted to a > kml but get the same message (albeit with a smaller byte size), and the > size of that file is 165KB. > > Does anyone know for sure if size is the issue here and if there is a way > (and how) to increase size limits for files to create geometry? Thanks > much. > > Andy > > -- > -- To post, send email to [email protected]. To unsubscribe, > send email to [email protected]. For more > information, visit https://groups.google.com/d/forum/archesproject?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Arches Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > > -- > -- To post, send email to [email protected]. To unsubscribe, > send email to [email protected]. For more > information, visit https://groups.google.com/d/forum/archesproject?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Arches Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- To post, send email to [email protected]. To unsubscribe, send email to [email protected]. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
