[GitHub] [incubator-sdap-nexus] skorper opened a new pull request #129: SDAP-312: Improved UTM reprojection speed

[GitBox]

skorper opened a new pull request #129:
URL: https://github.com/apache/incubator-sdap-nexus/pull/129


   I noticed the following pyproj call was deprecated:
   
   ```python
   pyproj.transform(p1=lonlat_proj, p2=aeqd_proj, x=x, y=y)
   ```
   
   I updated this call to the following:
   
   ```python
   aeqd_proj(x, y)
   ```
   
   which no longer throws a deprecation warning. I also noticed it's 
significantly faster. Here's a minimal example I threw together:
   
   ```python
   matchup_center_lon=-86.36720670954534
   matchup_center_lat=19.85014715590842
   y=22.1
   x=-94.0
   
   lonlat_proj = pyproj.Proj(proj='lonlat')
   aeqd_proj = pyproj.Proj(proj='aeqd', lon_0=matchup_center_lon, 
lat_0=matchup_center_lat)
   
   
   pyproj.transform(p1=lonlat_proj, p2=aeqd_proj, x=x, y=y)
   aeqd_proj(x, y)
   ```
   
   Both return the same result, but the first transform takes 0.073577404 
seconds and the second takes 0.000022650 seconds.
   
   I integrated this with the matchup algorithm and here are some example 
request times before/after:
   
   ```
   
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-09-30T23:59:59Z&tt=86400&rt=1000&b=-45,15,-30,30&platforms=1,2,3,4,5,6,7,8,9&parameter=sss&depthMin=0&depthMax=5&matchOnce=true&matchup=spurs
   ```
   Before: 5.23s
   After: 1.01s
   
   ```
   
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-10-25T00:00:00Z&tt=86400&rt=1000&b=-45,15,-30,30&platforms=1,2,3,4,5,6,7,8,9&parameter=sst&depthMin=0&depthMax=5&matchOnce=true&matchup=spurs
   ```
   Before: 13.35s
   After: 2.79s
   
   ```
   
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-10-25T00:00:00Z&tt=86400&rt=1000&b=-100,0,-30,30&platforms=1,2,3,4,5,6,7,8,9&parameter=sst&depthMin=0&depthMax=5&matchOnce=false&matchup=spurs
   ```
   Before: 67.72s
   After: 23.82s
   
   ```
   
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-12-25T00:00:00Z&tt=86400&rt=1000&b=-45,15,-30,30&platforms=1,2,3,4,5,6,7,8,9&parameter=sst&depthMin=0&depthMax=5&matchOnce=false&matchup=spurs
   ```
   Before: 41.91s
   After: 9.34s
   
   ```
   
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-09-30T23:59:59Z&tt=86400&rt=1000&b=-100,0,-30,30&platforms=1,2,3,4,5,6,7,8,9&parameter=sst&depthMin=0&depthMax=5&matchOnce=false&matchup=spurs
   ```
   Before: 20.34s
   After: 1.94s
   
   Here are the before/after metrics graphs:
   
   |Before|After|
   |--|--|
   
|![total_time](https://user-images.githubusercontent.com/11022336/119046176-3cf40480-b971-11eb-922d-ddf0413f4786.png)|![total_time_after](https://user-images.githubusercontent.com/11022336/119046205-467d6c80-b971-11eb-9886-180cba0eab89.png)|
   
|![1x_before](https://user-images.githubusercontent.com/11022336/119046734-e9ce8180-b971-11eb-806d-9f5ab7781bfa.png)|![1x_after](https://user-images.githubusercontent.com/11022336/119046751-f05cf900-b971-11eb-8e52-61db5edddd67.png)|
   
|![1x_tile_before](https://user-images.githubusercontent.com/11022336/119046781-f81c9d80-b971-11eb-9057-5bdfe4e36fa4.png)|![1x_tile_after](https://user-images.githubusercontent.com/11022336/119046815-ffdc4200-b971-11eb-9d57-cec7b1db8cef.png)|
   
|![num_tiles](https://user-images.githubusercontent.com/11022336/119046856-08cd1380-b972-11eb-8aff-1089f7f25b8d.png)|![num_tiles](https://user-images.githubusercontent.com/11022336/119046856-08cd1380-b972-11eb-8aff-1089f7f25b8d.png)|
   
   **Note: total time (as indicated in the graphs above and the before/after 
times above are the sum of the captured metrics, not the client total request 
time.**


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org