Re: [EXTERNAL] Re: Ex: [DISCUSS] Towards release 0.4.5a56

[Kuttruff, Riley K (US 398F-Affiliate)]

I think the granule ingester and collection manager have some common code 
components so it might make more sense for something like:
-apache-sdapv1.0.0
   -sdap-nexus-webapp
  -src
   -sdap-ingester
  -common
 -src
  -sdap-collection-manager
 -src
  -sdap-granule-ingester
 -src

On 11/2/22, 4:14 PM, "Frank Greguska"  wrote:

I believe we settled on 3 artifacts:

apache/sdap-nexus-webapp
apache/sdap-collection-manager
apache/sdap-granule-ingester

There was still some active discussion on if these should share a version
number or not. For sake of argument, let's say they do.

So tarball would be something like

   - apache-sdapv1.0.0
  - sdap-nexus-webapp
 - source code
 - sdap-collection-manager
 - source code
 - sdap-granule-ingester
 - source code


Github repos would all have v1.0.0 tags/releases
Docker repos would all have v1.0.0 tag

On Wed, Nov 2, 2022 at 3:52 PM Riley K. Kuttruff  wrote:

> Will there be a tarball corresponding to the Nexus & Ingester repos or
> will both be consolidated into one tatball?
>
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for 
Android
> 
> From: Perez, Stepheny K (US 398F) 
> Sent: Wednesday, November 2, 2022 3:47:25 PM
> To: dev@sdap.apache.org 
> Subject: Ex: [DISCUSS] Towards release 0.4.5a56
>
> Hi everyone,
>
> I’d like to start the conversation regarding our first official Apache
> release. From what I can tell, these are the major items that need to be
> completed before we can move forward:
>
>
> 1.   Identify a release manager. Any volunteers?
>
> 2.   Create release tarball
>
> 3.   Write release notes
>
> 4.   Write installation instructions from source (Riley Kuttruff has
> started this work)
>
> 5.   Push docker images to Dockerhub (and update quickstart with these
> versions)
>
> Another important task before moving forward would be testing the latest
> SDAP analysis image 0.4.5a56. I have personally used this version without
> any issues. Has anyone else upgraded to this latest alpha version?
>
> Thank you!
> Stepheny
> CAUTION: This email was not sent from a Cal Poly Pomona service. Exercise
> caution when clicking links or opening attachments. Please forward
> suspicious email to suspectem...@cpp.edu.
>
>

Re: Ex: [DISCUSS] Towards release 0.4.5a56

[Frank Greguska]

I believe we settled on 3 artifacts:

apache/sdap-nexus-webapp
apache/sdap-collection-manager
apache/sdap-granule-ingester

There was still some active discussion on if these should share a version
number or not. For sake of argument, let's say they do.

So tarball would be something like

   - apache-sdapv1.0.0
  - sdap-nexus-webapp
 - source code
 - sdap-collection-manager
 - source code
 - sdap-granule-ingester
 - source code


Github repos would all have v1.0.0 tags/releases
Docker repos would all have v1.0.0 tag

On Wed, Nov 2, 2022 at 3:52 PM Riley K. Kuttruff  wrote:

> Will there be a tarball corresponding to the Nexus & Ingester repos or
> will both be consolidated into one tatball?
>
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for Android
> 
> From: Perez, Stepheny K (US 398F) 
> Sent: Wednesday, November 2, 2022 3:47:25 PM
> To: dev@sdap.apache.org 
> Subject: Ex: [DISCUSS] Towards release 0.4.5a56
>
> Hi everyone,
>
> I’d like to start the conversation regarding our first official Apache
> release. From what I can tell, these are the major items that need to be
> completed before we can move forward:
>
>
> 1.   Identify a release manager. Any volunteers?
>
> 2.   Create release tarball
>
> 3.   Write release notes
>
> 4.   Write installation instructions from source (Riley Kuttruff has
> started this work)
>
> 5.   Push docker images to Dockerhub (and update quickstart with these
> versions)
>
> Another important task before moving forward would be testing the latest
> SDAP analysis image 0.4.5a56. I have personally used this version without
> any issues. Has anyone else upgraded to this latest alpha version?
>
> Thank you!
> Stepheny
> CAUTION: This email was not sent from a Cal Poly Pomona service. Exercise
> caution when clicking links or opening attachments. Please forward
> suspicious email to suspectem...@cpp.edu.
>
>

Re: Ex: [DISCUSS] Towards release 0.4.5a56

[Riley K. Kuttruff]

Will there be a tarball corresponding to the Nexus & Ingester repos or will 
both be consolidated into one tatball?

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android

From: Perez, Stepheny K (US 398F) 
Sent: Wednesday, November 2, 2022 3:47:25 PM
To: dev@sdap.apache.org 
Subject: Ex: [DISCUSS] Towards release 0.4.5a56

Hi everyone,

I’d like to start the conversation regarding our first official Apache release. 
From what I can tell, these are the major items that need to be completed 
before we can move forward:


1.   Identify a release manager. Any volunteers?

2.   Create release tarball

3.   Write release notes

4.   Write installation instructions from source (Riley Kuttruff has 
started this work)

5.   Push docker images to Dockerhub (and update quickstart with these 
versions)

Another important task before moving forward would be testing the latest SDAP 
analysis image 0.4.5a56. I have personally used this version without any 
issues. Has anyone else upgraded to this latest alpha version?

Thank you!
Stepheny
CAUTION: This email was not sent from a Cal Poly Pomona service. Exercise 
caution when clicking links or opening attachments. Please forward suspicious 
email to suspectem...@cpp.edu.

[DISCUSS] Towards release 0.4.5a56

[Perez, Stepheny K (US 398F)]

Hi everyone,

I’d like to start the conversation regarding our first official Apache release. 
From what I can tell, these are the major items that need to be completed 
before we can move forward:


1.   Identify a release manager. Any volunteers?

2.   Create release tarball

3.   Write release notes

4.   Write installation instructions from source (Riley Kuttruff has 
started this work)

5.   Push docker images to Dockerhub (and update quickstart with these 
versions)

Another important task before moving forward would be testing the latest SDAP 
analysis image 0.4.5a56. I have personally used this version without any 
issues. Has anyone else upgraded to this latest alpha version?

Thank you!
Stepheny

[GitHub] [incubator-sdap-ingester] RKuttruff opened a new pull request, #61: Improvements to ingestion

[GitBox]

RKuttruff opened a new pull request, #61:
URL: https://github.com/apache/incubator-sdap-ingester/pull/61

   This PR has several improvements to the granule ingester
   
   1. Made the writes to data and metadata stores more fault tolerant. In the 
existing implementation, we write to the underlying stores very frequently. 
This seems to overburden them, and eventually a write may fail. This was 
treated as an unrecoverable loss of connection and the GI fails with the 
granule needing to be restarted from the beginning. This PR has the writes 
retry a few times on failure with some backoff time.
   2. Consolidated writes. For small tile sizes (and thus a large number of 
tiles/granule), there seems to be a lot of time spent writing the data to the 
underlying stores. This PR consolidates them into a large push after all the 
tiles have been generated, significantly reducing the number of network I/O 
operations.
   3. Optimized data subset conversion to `np.ndarray`. Initially done through 
a call to `np.ma.filled` with a `list` of `xr.DataArrays`. This is unnecessary 
because a) `np.ma.filled` is supposed to be used with masked arrays, and with 
`list` objects, just calls `np.array`; and b) there seems to be a lot on 
inefficiency with calling `np.array` with `xr.DataArrays`, especially when the 
`xr.DataArray` objects are backed by `np.ndarray` objects that can be easily 
fetched from them. This PR does just that.
   
   With these changes listed above, a benchmark ingest of 1 VIIRS granule into 
100x100 tiles went from 4580 seconds to just 163 seconds.
   
   Further testing + benchmarking coming soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [EXTERNAL] November Apache SDAP report draft

[Perez, Stepheny K (US 398F)]

I removed the release from the report. I’ll also start that conversation 
regarding release 0.4.5a56. Thanks again!

Stepheny

From: Julian Hyde 
Date: Wednesday, November 2, 2022 at 1:20 PM
To: dev@sdap.apache.org 
Subject: Re: [EXTERNAL] November Apache SDAP report draft
It’s not a release for Apache’s purposes because it hasn’t been through the 
signing/vote/publish/announce process. An Apache release is a legal and 
community action more than it is a piece of code. The software quality of the 
release (e.g. ‘alpha’) is irrelevant to Apache, but the process of creating the 
release is all-important.

Yes, I think you should remove the release from the report. If you do that I’ll 
amend my comments.

I am happy to work with SDAP committers on an Apache-style release. We can 
probably use the 0.4.5a56 tag as a starting point. A good starting point would 
be an email to the dev list with the subject “[DISCUSS] Towards release 
0.4.5a56” asking when would be a good time for a release, volunteer for release 
manager, agree on a list of tasks, etc.

Julian




> On Nov 2, 2022, at 11:38 AM, Perez, Stepheny K (US 398F) 
>  wrote:
>
> Julian,
>
> The release I was referring to in our report was 0.4.5a56 which is an alpha 
> release. Would specifying this in the report make it more accurate? If not, I 
> can remove that from the report altogether. Thanks for your feedback.
>
> Stepheny
>
> From: Julian Hyde 
> Date: Wednesday, November 2, 2022 at 11:22 AM
> To: dev@sdap.apache.org 
> Subject: [EXTERNAL] Re: November Apache SDAP report draft
> Thanks for writing and submitting the report, Stepheny.
>
> I signed off and made the following comments:
>
>> The report says that there has been a release. I dispute
>> that. There is no release vote on dev@, no mention of the
>> release on dev@, nothing on announce@, and no downloads page.
>>
>> As far as I can tell, this project has never made an Apache
>> release.
>>
>> SDAP's number 1 problem is their use of dev@. There is activity
>> in the project (including commit messages going to dev@) but
>> questions asked on dev@ go unanswered, and therefore
>> conversations don't happen. Even the report is drafted on
>> private@ rather than dev@.  I think I can work with SDAP and get
>> them to graduate but only if they engage on dev@.
>>
>> I have given notice that I will resign as a mentor by Dec 31st if
>> I don't see activity, namely a plan to graduate, and activity to
>> carry out that plan.
>
> Julian
>
>
> On 2022/11/02 00:57:44 "Perez, Stepheny K (US 398F)" wrote:
>> Hello,
>>
>> Attached you will find a draft of the November 2022 Apache SDAP report. In 
>> this report, I’ve included details about what steps we still need to take 
>> before graduating and a target date for graduating. Please provide feedback 
>> on the contents, as I’ll be submitting the reporting today at C.O.B.
>>
>> Thank you!
>> Stepheny
>>
>>

Re: [EXTERNAL] November Apache SDAP report draft

[Julian Hyde]

It’s not a release for Apache’s purposes because it hasn’t been through the 
signing/vote/publish/announce process. An Apache release is a legal and 
community action more than it is a piece of code. The software quality of the 
release (e.g. ‘alpha’) is irrelevant to Apache, but the process of creating the 
release is all-important.

Yes, I think you should remove the release from the report. If you do that I’ll 
amend my comments.

I am happy to work with SDAP committers on an Apache-style release. We can 
probably use the 0.4.5a56 tag as a starting point. A good starting point would 
be an email to the dev list with the subject “[DISCUSS] Towards release 
0.4.5a56” asking when would be a good time for a release, volunteer for release 
manager, agree on a list of tasks, etc.

Julian




> On Nov 2, 2022, at 11:38 AM, Perez, Stepheny K (US 398F) 
>  wrote:
> 
> Julian,
> 
> The release I was referring to in our report was 0.4.5a56 which is an alpha 
> release. Would specifying this in the report make it more accurate? If not, I 
> can remove that from the report altogether. Thanks for your feedback.
> 
> Stepheny
> 
> From: Julian Hyde 
> Date: Wednesday, November 2, 2022 at 11:22 AM
> To: dev@sdap.apache.org 
> Subject: [EXTERNAL] Re: November Apache SDAP report draft
> Thanks for writing and submitting the report, Stepheny.
> 
> I signed off and made the following comments:
> 
>> The report says that there has been a release. I dispute
>> that. There is no release vote on dev@, no mention of the
>> release on dev@, nothing on announce@, and no downloads page.
>> 
>> As far as I can tell, this project has never made an Apache
>> release.
>> 
>> SDAP's number 1 problem is their use of dev@. There is activity
>> in the project (including commit messages going to dev@) but
>> questions asked on dev@ go unanswered, and therefore
>> conversations don't happen. Even the report is drafted on
>> private@ rather than dev@.  I think I can work with SDAP and get
>> them to graduate but only if they engage on dev@.
>> 
>> I have given notice that I will resign as a mentor by Dec 31st if
>> I don't see activity, namely a plan to graduate, and activity to
>> carry out that plan.
> 
> Julian
> 
> 
> On 2022/11/02 00:57:44 "Perez, Stepheny K (US 398F)" wrote:
>> Hello,
>> 
>> Attached you will find a draft of the November 2022 Apache SDAP report. In 
>> this report, I’ve included details about what steps we still need to take 
>> before graduating and a target date for graduating. Please provide feedback 
>> on the contents, as I’ll be submitting the reporting today at C.O.B.
>> 
>> Thank you!
>> Stepheny
>> 
>> 

Re: [EXTERNAL] Re: November Apache SDAP report draft

[Perez, Stepheny K (US 398F)]

Julian,

The release I was referring to in our report was 0.4.5a56 which is an alpha 
release. Would specifying this in the report make it more accurate? If not, I 
can remove that from the report altogether. Thanks for your feedback.

Stepheny

From: Julian Hyde 
Date: Wednesday, November 2, 2022 at 11:22 AM
To: dev@sdap.apache.org 
Subject: [EXTERNAL] Re: November Apache SDAP report draft
Thanks for writing and submitting the report, Stepheny.

I signed off and made the following comments:

> The report says that there has been a release. I dispute
> that. There is no release vote on dev@, no mention of the
> release on dev@, nothing on announce@, and no downloads page.
>
> As far as I can tell, this project has never made an Apache
> release.
>
> SDAP's number 1 problem is their use of dev@. There is activity
> in the project (including commit messages going to dev@) but
> questions asked on dev@ go unanswered, and therefore
> conversations don't happen. Even the report is drafted on
> private@ rather than dev@.  I think I can work with SDAP and get
> them to graduate but only if they engage on dev@.
>
> I have given notice that I will resign as a mentor by Dec 31st if
> I don't see activity, namely a plan to graduate, and activity to
> carry out that plan.

Julian


On 2022/11/02 00:57:44 "Perez, Stepheny K (US 398F)" wrote:
> Hello,
>
> Attached you will find a draft of the November 2022 Apache SDAP report. In 
> this report, I’ve included details about what steps we still need to take 
> before graduating and a target date for graduating. Please provide feedback 
> on the contents, as I’ll be submitting the reporting today at C.O.B.
>
> Thank you!
> Stepheny
>
>

Re: November Apache SDAP report draft

[Julian Hyde]

Thanks for writing and submitting the report, Stepheny.

I signed off and made the following comments:

> The report says that there has been a release. I dispute
> that. There is no release vote on dev@, no mention of the
> release on dev@, nothing on announce@, and no downloads page.
>
> As far as I can tell, this project has never made an Apache
> release.
>
> SDAP's number 1 problem is their use of dev@. There is activity
> in the project (including commit messages going to dev@) but
> questions asked on dev@ go unanswered, and therefore
> conversations don't happen. Even the report is drafted on
> private@ rather than dev@.  I think I can work with SDAP and get
> them to graduate but only if they engage on dev@.
>
> I have given notice that I will resign as a mentor by Dec 31st if
> I don't see activity, namely a plan to graduate, and activity to
> carry out that plan.

Julian


On 2022/11/02 00:57:44 "Perez, Stepheny K (US 398F)" wrote:
> Hello,
> 
> Attached you will find a draft of the November 2022 Apache SDAP report. In 
> this report, I’ve included details about what steps we still need to take 
> before graduating and a target date for graduating. Please provide feedback 
> on the contents, as I’ll be submitting the reporting today at C.O.B.
> 
> Thank you!
> Stepheny
> 
>