Thanks to both the Adams for input. Just as background these gpx files are the tracklines for our field teams over the course of an entire day, when you are collecting a fix every second it doesn't take long for the file sizes to get large, especially when it appears that we can only upload something in the ball park of 130KBs.
Our plan is to have the teams save the full gpx file as a Media resource which others could download if need be, but also add the gpx as geometry for a survey so someone can quickly see the general location the team was in by just looking at the resource report (without downloading the full gpx file). I think the best solution for us is to have teams edit their own gpx files in a 3rd party software (such as Garmin BaseCamp) to remove unnecessary points, and also save the file after filtering/downsampling to create a smaller file. This keeps the data flow in the hand of the people who collected the data and only adds an additional step (filtering, as they would need to edit the files anyway to remove extra points). I think I also prefer to filter as opposed to decreasing precision. Decreasing precision could potential show the team being in a location they never visited (albeit fairly close to where they actually were depending on the number of digits deleted), while with filtering points will be lost but the points that do show up will be in the exact spot that the team was. Probably a moot point as this is all just to provide a general overview of where the team was. Anyway, long way of saying thanks for the info and suggestions, I think we have work around to our issue. Andy On Monday, May 2, 2016 at 8:04:21 AM UTC-7, Adam Cox wrote: > > Hi Andy, both .gpx and .kml files are written in xml, so you could open > them and read it with notepad++, or with a simple python script, and use > some regex or something to trim all of the coordinates to fewer decimal > places. That would be a good way to incorporate this into your workflow... > > On Sun, May 1, 2016 at 4:38 PM, Adam Lodge <[email protected] > <javascript:>> wrote: > >> Andy, >> >> Happy to help, but unfortunately, we don’t have a solution in our hip >> pocket. We have only encountered this kind of issue when transferring >> larger numbers of records from legacy systems into Arches. Therefore, our >> solution is a bit one-off. And, I am a bit surprised that geometry (that I >> assume was) collected with GPS units would be voluminous enough to surpass >> that threshold. >> >> That said, what I have done in the past is run the data through an >> appropriate FME transformer or use Postgis (st_snaptogrid function) to >> process the data in a manner that resolves the issue. >> >> Starting to move beyond my domain of expertise, but it seems likely that >> there is a gdal function that you could bind to your python ETL/integration >> code that may do the trick. Wish I could point you to something more >> specific. Maybe someone else can? >> >> One other thing - I highly recommend you start with reducing precision >> rather than generalizing. You preserve much more integrity of the data and >> still likely achieve your goal. >> >> Adam >> >> -- >> Adam Lodge >> Geospatial Systems Consultant >> Farallon Geographics >> >> On Sunday, May 1, 2016 at 3:16 PM, Andy Graham wrote: >> >> Thanks for the info Adam. I figured it was something like that. In this >> case generalizing the gpx probably is acceptable. Do you guys know of or >> have a quick and easy program/code to do this? I can manage this with a >> few steps but I visualize this being part of the daily work flow so would >> prefer more of a single step, "push a button and it's generalized" type >> setup. Let me know. Thanks. >> >> Andy >> On May 1, 2016 2:17 PM, "Adam Lodge" <[email protected] <javascript:>> >> wrote: >> >> Andy, >> >> There is a known limitation - Im guessing from your error message that >> 32766 - to the number of characters that Elastic Search can process in a >> single value. This translates into a geometry limitation because (I think) >> ES indexes geometries as geojson. In short, I think it’s less about the >> size of the file and more about the number of characters necessary to >> describe the geometry. >> >> We have done some tricks to push geometries like these through. They >> include: >> - Reduce the precision of the captured geometries (thereby reducing the >> number of characters after the decimal that gets stored for each vertex) >> - Generalize the geometry (if that is acceptable.) That reduces the >> number of vertices in the geometry. >> >> If you are able to send the file, I would be happy to take a look at it. >> >> Adam >> >> -- >> Adam Lodge >> Geospatial Systems Consultant >> Farallon Geographics >> 415.317.6625 >> >> On Sunday, May 1, 2016 at 2:08 PM, Andy Graham wrote: >> >> So I have been trying to add geometry to one of our resources by dragging >> and dropping a .gpx file onto the map in the edit form for the resource. >> >> I can add it to the map just fine but when I go to save I eventually get >> an error on the webpage that states: >> >> TransportError at >> /resources/SURVEY.E1/default/1510042c-6888-4fc9-8658-cd5cee6a8bdb >> >> TransportError(500, u'IllegalArgumentException[Document contains at least >> one immense term in field="geometries.label" (whose UTF8 encoding is longer >> than the max length 32766), all of which were skipped. Please correct the >> analyzer to not produce such terms. The prefix of the first immense term >> is: \'[77, 85, 76, 84, 73, 76, 73, 78, 69, 83, 84, 82, 73, 78, 71, 40, 40, >> 45, 49, 50, 48, 46, 53, 54, 57, 57, 53, 49, 48, 51]...\', original message: >> bytes can be at most 32766 in length; got 100421]; nested: >> MaxBytesLengthExceededException[bytes can be at most 32766 in length; got >> 100421]; ') >> >> >> From what I can tell it appears as though there is a size limit on the >> files used to create the geometry? I have done this successfully with >> smaller gpx files, this current one is about 300KB. I have converted to a >> kml but get the same message (albeit with a smaller byte size), and the >> size of that file is 165KB. >> >> Does anyone know for sure if size is the issue here and if there is a way >> (and how) to increase size limits for files to create geometry? Thanks >> much. >> >> Andy >> >> -- >> -- To post, send email to [email protected] <javascript:>. To >> unsubscribe, send email to [email protected] <javascript:>. >> For more information, visit >> https://groups.google.com/d/forum/archesproject?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Arches Project" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> -- To post, send email to [email protected] <javascript:>. To >> unsubscribe, send email to [email protected] <javascript:>. >> For more information, visit >> https://groups.google.com/d/forum/archesproject?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Arches Project" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- -- To post, send email to [email protected]. To unsubscribe, send email to [email protected]. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
