skorper opened a new pull request, #202: URL: https://github.com/apache/incubator-sdap-nexus/pull/202
[SDAP-402](https://issues.apache.org/jira/browse/SDAP-402) - Fixed matchOnce logic. Previously, only a single secondary point was matched to a primary point. Now, we are allowing multiple matches so long as they share the same time and space as the closest point. - Fixed the failing matchup unit tests - Added a new unit test case for this change - Fixed bug where `depth` was being overwritten with a null value (maybe will fix our depth problems?) Tested this locally with the following request (from the google doc): ``` {{big_data_url}}/match_spark?tt=43200&primary=JPL-L4-MRVA-CHLA-GLOB-v3.0&secondary=shark-2018&startTime=2018-04-01T00%3A00%3A00Z&endTime=2018-04-01T12%3A59%3A59Z&rt=25000&b=-140%2C10%2C-110%2C40&platforms=3B&depthMin=-5&depthMax=5&matchOnce=true ``` <details> <summary>This results in the following json output:</summary> ```json { "executionId": "e4fb1b4f-c002-4c68-a651-94db87e96c2f", "data": [ { "platform": "orbiting satellite", "device": "radiometers", "lon": "-129.875", "lat": "27.125", "point": "Point(-129.875 27.125)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 18, 20]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.0518328994512558, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "113", "lon": "-130.015296", "lat": "27.0813824", "point": "Point(-130.015296 27.0813824)", "time": 1522570620, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.11, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-130.015296", "lat": "27.0813824", "point": "Point(-130.015296 27.0813824)", "time": 1522570620, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.628, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 18.822, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-129.875", "lat": "27.375", "point": "Point(-129.875 27.375)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 19, 20]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.05334627255797386, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "102", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": -2.3, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "air_temperature", "cf_variable_name": "air_temperature", "variable_value": 18.58, "variable_unit": "C" }, { "variable_name": "air_temperature_quality", "cf_variable_name": "air_temperature_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "relative_humidity", "cf_variable_name": "relative_humidity", "variable_value": 60.88, "variable_unit": "1" }, { "variable_name": "relative_humidity_quality", "cf_variable_name": "relative_humidity_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "102", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": -0.2, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "air_pressure", "cf_variable_name": "air_pressure", "variable_value": 1019.2, "variable_unit": "hPa" }, { "variable_name": "air_pressure_quality", "cf_variable_name": "air_pressure_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "113", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.09, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.669, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 19.623, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-130.125", "lat": "27.375", "point": "Point(-130.125 27.375)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 19, 19]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.05735308676958084, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "102", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": -2.3, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "air_temperature", "cf_variable_name": "air_temperature", "variable_value": 18.58, "variable_unit": "C" }, { "variable_name": "air_temperature_quality", "cf_variable_name": "air_temperature_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "relative_humidity", "cf_variable_name": "relative_humidity", "variable_value": 60.88, "variable_unit": "1" }, { "variable_name": "relative_humidity_quality", "cf_variable_name": "relative_humidity_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "102", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": -0.2, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "air_pressure", "cf_variable_name": "air_pressure", "variable_value": 1019.2, "variable_unit": "hPa" }, { "variable_name": "air_pressure_quality", "cf_variable_name": "air_pressure_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "113", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.09, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-130.020352", "lat": "27.240928", "point": "Point(-130.020352 27.240928)", "time": 1522540800, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.669, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 19.623, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-130.125", "lat": "26.875", "point": "Point(-130.125 26.875)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 17, 19]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.05785427987575531, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "113", "lon": "-129.9747584", "lat": "26.8175136", "point": "Point(-129.9747584 26.8175136)", "time": 1522608780, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.08, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-129.9747584", "lat": "26.8175136", "point": "Point(-129.9747584 26.8175136)", "time": 1522608780, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.565, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 18.797, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-130.125", "lat": "27.125", "point": "Point(-130.125 27.125)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 18, 19]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.05617373436689377, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "113", "lon": "-130.0269312", "lat": "27.1240416", "point": "Point(-130.0269312 27.1240416)", "time": 1522564980, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.12, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-130.0269312", "lat": "27.1240416", "point": "Point(-130.0269312 27.1240416)", "time": 1522564980, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.618, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 18.847, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-130.125", "lat": "26.625", "point": "Point(-130.125 26.625)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 16, 19]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.05913911014795303, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "102", "lon": "-130.0557312", "lat": "26.6866656", "point": "Point(-130.0557312 26.6866656)", "time": 1522627200, "depth": -2.3, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "air_temperature", "cf_variable_name": "air_temperature", "variable_value": 18.09, "variable_unit": "C" }, { "variable_name": "air_temperature_quality", "cf_variable_name": "air_temperature_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "relative_humidity", "cf_variable_name": "relative_humidity", "variable_value": 66.92, "variable_unit": "1" }, { "variable_name": "relative_humidity_quality", "cf_variable_name": "relative_humidity_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "102", "lon": "-130.0557312", "lat": "26.6866656", "point": "Point(-130.0557312 26.6866656)", "time": 1522627200, "depth": -0.2, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "air_pressure", "cf_variable_name": "air_pressure", "variable_value": 1018.12, "variable_unit": "hPa" }, { "variable_name": "air_pressure_quality", "cf_variable_name": "air_pressure_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "113", "lon": "-130.0557312", "lat": "26.6866656", "point": "Point(-130.0557312 26.6866656)", "time": 1522627200, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.08, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-130.0557312", "lat": "26.6866656", "point": "Point(-130.0557312 26.6866656)", "time": 1522627200, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.624, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 19.177, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-129.875", "lat": "26.875", "point": "Point(-129.875 26.875)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 17, 20]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.05762673169374466, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "113", "lon": "-129.9238912", "lat": "26.8958944", "point": "Point(-129.9238912 26.8958944)", "time": 1522597380, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.12, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-129.9238912", "lat": "26.8958944", "point": "Point(-129.9238912 26.8958944)", "time": 1522597380, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.677, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 18.909, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] }, { "platform": "orbiting satellite", "device": "radiometers", "lon": "-129.875", "lat": "26.625", "point": "Point(-129.875 26.625)", "time": 1522584000, "depth": null, "fileurl": "20180401120000-JPL-CHL25-fv03.nc", "id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 16, 20]]", "source": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "primary": [ { "variable_name": "CHLA_analysis", "cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water", "variable_value": 0.061026107519865036, "variable_unit": null } ], "matches": [ { "platform": "3B", "device": "113", "lon": "-130.0324224", "lat": "26.721784", "point": "Point(-130.0324224 26.721784)", "time": 1522621860, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water", "variable_value": 0.08, "variable_unit": "kg m-3" }, { "variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality", "variable_value": 2, "variable_unit": null } ] }, { "platform": "3B", "device": "130", "lon": "-130.0324224", "lat": "26.721784", "point": "Point(-130.0324224 26.721784)", "time": 1522621860, "depth": 0.5, "fileurl": null, "id": null, "source": "shark-2018", "secondary": [ { "variable_name": "sea_water_practical_salinity", "cf_variable_name": "sea_water_practical_salinity", "variable_value": 34.631, "variable_unit": "1" }, { "variable_name": "sea_water_practical_salinity_quality", "cf_variable_name": "sea_water_practical_salinity_quality", "variable_value": 2, "variable_unit": null }, { "variable_name": "sea_water_temperature", "cf_variable_name": "sea_water_temperature", "variable_value": 19.126, "variable_unit": "C" }, { "variable_name": "sea_water_temperature_quality", "cf_variable_name": "sea_water_temperature_quality", "variable_value": 2, "variable_unit": null } ] } ] } ], "params": { "primary": "JPL-L4-MRVA-CHLA-GLOB-v3.0", "matchup": "shark-2018", "startTime": 1522540800, "endTime": 1522587599, "bbox": "-140,10,-110,40", "timeTolerance": 43200, "radiusTolerance": 25000.0, "platforms": "3B", "parameter": null, "depthMin": -5.0, "depthMax": 5.0 }, "bounds": {}, "count": 8, "details": { "timeToComplete": 119, "numSecondaryMatched": 22, "numPrimaryMatched": 8 } } ``` </details> Notice how there are 8 primary matches but 22 secondary matches. If we look at one of the matches, we can see the secondary points are at the same time/space as all the secondary matches. When running the above query on the old algorithm we get the following counts: `matchOnce=true`: ```json { "timeToComplete": 39, "numSecondaryMatched": 8, "numPrimaryMatched": 8 } ``` `matchOnce=false`: ```json { "timeToComplete": 32, "numSecondaryMatched": 10784, "numPrimaryMatched": 8 } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
