Good afternoon! Thank you again for yesterday's meeting! Reaching out again in regard to setting up a follow-up meeting! Once the meeting time is confirmed, we’ll provide more details regarding the issues we’re encountering as well as the progress we’ve made. To facilitate scheduling, please use the following Doodle link:
https://doodle.com/meeting/participate/id/e1QmZlme Regarding the issue we discussed yesterday—while all containers are running, Cassandra is not connecting with the Granule Ingester despite indications of processing. After multiple attempts to pinpoint the problem, we’re still coming up short. We did try local build but encountered new issues there as well. For context, we’re primarily focusing on MacOS for the time being. Lastly, a question for Nga Chung: When testing version 1.4.0, was there any deviation from the Quick Start guide? Were there alternative procedures used to test the new release, or were any regression tests conducted specifically for the new releases? Looking forward to your thoughts and the opportunity to connect again soon. Best regards, 1. Marc Cruz ________________________________ From: Riley Kuttruff <r...@apache.org> Sent: Tuesday, November 19, 2024 3:53 PM To: Marc A. Cruz <marcc...@cpp.edu> Cc: Truc N. Nguyen <trucngu...@cpp.edu>; Riley Kuttruff <r...@apache.org>; dev@sdap.apache.org <dev@sdap.apache.org> Subject: Re: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team You don't often get email from r...@apache.org. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> No worries. Please let us know how you progress with your work or if you encounter any more issues! -Riley On Tue, Nov 19, 2024 at 2:47 PM Marc A. Cruz <marcc...@cpp.edu<mailto:marcc...@cpp.edu>> wrote: Good afternoon, I appreciate this morning's meeting to help troubleshoot the problems we had! My partner and I wanted to let you know that we cannot attend the community meeting today. We will continue working on this and will keep you posted. Thank you for your time, Marc Cruz ________________________________ From: Truc N. Nguyen <trucngu...@cpp.edu<mailto:trucngu...@cpp.edu>> Sent: Monday, November 18, 2024 10:23:51 AM To: Riley Kuttruff <r...@apache.org<mailto:r...@apache.org>>; Marc A. Cruz <marcc...@cpp.edu<mailto:marcc...@cpp.edu>> Subject: Re: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team Hi! Thank you for letting us know. Just did it. Sincerely, Truc Nguyen California State Polytechnic University, Pomona trucngu...@cpp.edu<mailto:trucngu...@cpp.edu> Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Riley Kuttruff <r...@apache.org<mailto:r...@apache.org>> Sent: Monday, November 18, 2024 10:20:05 AM To: Truc N. Nguyen <trucngu...@cpp.edu<mailto:trucngu...@cpp.edu>>; Marc A. Cruz <marcc...@cpp.edu<mailto:marcc...@cpp.edu>> Subject: Fwd: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team You don't often get email from r...@apache.org<mailto:r...@apache.org>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi, I'm forwarding this response directly since I'm not sure if you've subscribed to the SDAP mailing list which is what I sent my reply to initially. If you want to subscribe, send an email here<mailto:dev-subscr...@sdap.apache.org?subject=subscribe>. I don't think you need to put anything in the message body. You should get a reply to confirm. ---------- Forwarded message --------- From: Riley Kuttruff <r...@apache.org<mailto:r...@apache.org>> Date: Mon, Nov 18, 2024 at 9:48 AM Subject: Re: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team To: <dev@sdap.apache.org<mailto:dev@sdap.apache.org>> Thanks and sorry you ran into so many issues with the quickstart - I wrote it a couple years ago as an intern and have only recreated the Cassandra/ZK/Solr/RMQ containers once (generally, you can just stop them and restart them no problem). I've been seeing more and more issues with using host.docker.internal / localhost and, given that using the container name as the hostname seems to be what needs to be done, you may wish to propose that as a change to the documentation. For the missing data in Jupyter: have you verified the data completed ingestion? You can check in the logs of the ingester containers. You can also open Solr admin console at localhost:8983 in your browser, select the nexustiles collection in the dropdown menu on the left, and query all tiles to verify they are there. For the documentation questions: the incubator-sdap-nexus RTD is obsolete. It's from when we were in ASF Incubator and we are unable to edit the RTD project subdomain so we migrated to sdap-nexus. I believe there should be a way to set it up to redirect, but we must have forgotten to configure that. Finally you do not need kubernetes to run the quickstart, though if you wish to try deploying to k8s, we have a Helm chart though I've always had to go through a bit of troubleshooting when deploying it fresh (that said, the k8s clusters I've deployed to were locked down permissions-wise). You asked about the code block of version tags: "With the code block of all version tags, would the given work? As we are not familiar with how sensitive each container is to new changes found in other containers." What is this in reference to? If you posted a screenshot, the ASF mailing lists do not support images so you'll need to paste plain text. Sorry if I misunderstood what you're asking there though. Finally, I'm definitely concerned about the crashes on Debian 12. I've never used it myself (I've really only used MacOS for running this - outside of k8s that is) but we should definitely be ensuring that we can run on common platforms. -Riley On Mon, Nov 18, 2024 at 9:26 AM Nga Chung <nch...@apache.org<mailto:nch...@apache.org>> wrote: > > Forwarding to dev@sdap > > ---------- Forwarded message --------- > From: Truc N. Nguyen <trucngu...@cpp.edu<mailto:trucngu...@cpp.edu>> > Date: Mon, Nov 18, 2024 at 12:43 AM > Subject: Re: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting > with SDAP Team > To: Nga Chung <nch...@apache.org<mailto:nch...@apache.org>>, Marc A. Cruz > <marcc...@cpp.edu<mailto:marcc...@cpp.edu>> > > > Good morning SDAP team, > > Thank you for your quick response! For the issues, here is the trail > we blazed thus far while working through the SDAP QuickStart process, > as well as a few questions regarding the documentation. > Progress and Challenges > > For myself: > > RabbitMQ issues: > I encountered an error (ACCESS_REFUSED - Login was refused using > authentication mechanism PLAIN) when trying to run the two ingester > containers on macOS. I resolved this by modifying RABBITMQ_HOST to rmq > in the granule-ingester.env file, though I’m unsure if this is the > recommended solution. > > Jupyter Notebook Data Issue: > While I successfully got the Jupyter Notebook running, I encountered a > problem where the data wasn’t found when running the Time Series > Examples, despite verifying that the data had been downloaded > correctly. > > For Marc, my partner: > > Initially, we attempted to use version 1.3.0 of the Docker images, but > documentation references version 1.2.0. Due to issues with older > images not being available, we proceeded with version 1.4.0. > Encountered Docker Desktop issues on Windows, prompting a switch to > macOS and Debian 12. > > For Debian 12: > > The granule ingester failed to connect to Cassandra, requiring us to > troubleshoot by replacing host.docker.internal to localhost ,or other > container-specific names (e.g., solr for History URL in connection > manager). > Updated the granule-ingester.env file as follows: > > CASSANDRA_CONTACT_POINTS=172.18.0.3 > CASSANDRA_USERNAME=cassandra > CASSANDRA_PASSWORD=cassandra > Here, the CASSANDRA_CONTACT_POINTS value was manually set to the > container’s IP address using docker inspect cassandra. > Created a new RabbitMQ user with admin access and set the virtual host to /. > > After getting granule ingester to not abort due to connection with > cassandra and rabbitmq. collection-manager was able to run and then > Debian 12 docker desktop crashed. Tried with default settings for > resources and also near maximum resources. Both cases of docker > desktop configuration crashed. We apologize that we can not provide > too many details of the crash as Docker Desktop premium is required to > determine exact causes of the crashes. > > To conclude, with all the mentioned troubleshooting, I was able to > complete all the steps then a crash occurs. > > > Questions Regarding Documentation > > For documentation, there seems to be two > sdap-nexus.readthedocs.io<http://sdap-nexus.readthedocs.io/> & > incubator-sdap-nexus readthedocs.io<http://readthedocs.io/>. Can you confirm > whether these are > the same or serve different purposes? > > For example, > https://sdap-nexus.readthedocs.io/en/1.4.0/quickstart.html > export COLLECTION_MANAGER_VERSION=1.4.0 > export GRANULE_INGESTER_VERSION=1.4.0 > > https://incubator-sdap-nexus.readthedocs.io/en/latest/quickstart.html > export COLLECTION_MANAGER_VERSION=1.2.0 > export GRANULE_INGESTER_VERSION=1.2.0 > > With the code block of all version tags, would the given work? As we > are not familiar with how sensitive each container is to new changes > found in other containers. > > Was Kubernetes a pre-requisite to run docker QuickStart? > > > Thank you for the help and support, we also filled out the doodle for > meeting time so we're looking forward to seeing everyone! > > Thank you for your time. > > Sincerely, > Truc Nguyen > California State Polytechnic University, Pomona > trucngu...@cpp.edu<mailto:trucngu...@cpp.edu> > ________________________________ > From: Nga Chung <nch...@apache.org<mailto:nch...@apache.org>> > Sent: Sunday, November 17, 2024 9:11 PM > To: Truc N. Nguyen <trucngu...@cpp.edu<mailto:trucngu...@cpp.edu>>; Marc A. > Cruz <marcc...@cpp.edu<mailto:marcc...@cpp.edu>> > Subject: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team > > [You don't often get email from nch...@apache.org<mailto:nch...@apache.org>. > Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > Third time's the charm... Sorry by default when we reply to dev@ > mailing list it only includes dev@. > > Looking forward to speaking to you both. > > Best, > Nga > > On Sun, Nov 17, 2024 at 9:09 PM Nga Chung > <nch...@apache.org<mailto:nch...@apache.org>> wrote: > > > > Sorry, forget to reply all! Truc, please see my message below. > > > > Thanks, > > Nga > > > > > > > > On Sun, Nov 17, 2024 at 9:07 PM Nga Chung > > <nch...@apache.org<mailto:nch...@apache.org>> wrote: > > > > > > Hi Truc, > > > > > > We'd be happy to meet with you. As Riley mentioned, it would be great > > > if you can please provide a brief description of the challenges, so we > > > can come to the meeting prepared. > > > > > > I've also created a doodle poll, so for those who would like to join > > > us, please fill out the poll and we will find a time that works for > > > everyone. > > > > > > Link to poll: > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoodle.com%2Fmeeting%2Fparticipate%2Fid%2FaQvQZAlb&data=05%7C02%7Ctrucnguyen%40cpp.edu%7Cf93cdbddbf854256e15608dd078f846b%7C164ba61e39ec4f5d89ffaa1f00a521b4%7C0%7C0%7C638675035179090142%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JD0kfSP6L3poS3Vl1FbfJC4d0SisdY9lKjp60sWBL8U%3D&reserved=0<https://doodle.com/meeting/participate/id/aQvQZAlb> > > > > > > Thanks, > > > Nga > > > > > > On Fri, Nov 15, 2024 at 1:06 PM Riley Kuttruff > > > <r...@apache.org<mailto:r...@apache.org>> wrote: > > > > > > > > Hi! > > > > > > > > I'd be happy to help in any way I can. Could you perhaps provide a brief > > > > description of the technical challenges you've been facing? Just so I > > > > can > > > > have a little time to prepare to make the most of our time. > > > > > > > > I'm happy to meet next week though my calendar is a bit full so we may > > > > need > > > > to iterate a little to find a suitable time. What times work for you? > > > > > > > > Also, I think @nchung may also wish to attend. > > > > > > > > -Riley > > > > > > > > On Fri, Nov 15, 2024, 12:52 PM Truc N. Nguyen > > > > <trucngu...@cpp.edu<mailto:trucngu...@cpp.edu>> wrote: > > > > > > > > > Good afternoon, SDAP Team, > > > > > > > > > > My partner and I are students in Dr. Thomas Huang’s Parallel > > > > > Processing > > > > > class, and we are working on a project about Apache SDAP ETL pipeline > > > > > using > > > > > Apache Airflow as part of our class presentation. > > > > > > > > > > Dr. Huang recommended reaching out to your team for guidance. We are > > > > > currently facing some technical challenges and would greatly > > > > > appreciate any > > > > > support or advice you could provide. > > > > > > > > > > Would it be possible to schedule a meeting with your team sometime > > > > > next > > > > > week? We are also aware of your monthly meeting scheduled for next > > > > > Tuesday, > > > > > and while we’re unsure if we can attend, we will make an effort to > > > > > join if > > > > > it's possible. > > > > > > > > > > Thank you for your time and look forward to hearing from you soon. > > > > > > > > > > Sincerely, > > > > > Truc Nguyen > > > > > California State Polytechnic University, Pomona > > > > > trucngu...@cpp.edu<mailto:trucngu...@cpp.edu> > > > > > > CAUTION: This email was not sent from a Cal Poly Pomona service. > Exercise caution when clicking links or opening attachments. Please > forward suspicious email to > suspectem...@cpp.edu<mailto:suspectem...@cpp.edu><mailto:suspectem...@cpp.edu<mailto:suspectem...@cpp.edu>>. CAUTION: This email was not sent from a Cal Poly Pomona service. Exercise caution when clicking links or opening attachments. Please forward suspicious email to suspectem...@cpp.edu<mailto:suspectem...@cpp.edu>. CAUTION: This email was not sent from a Cal Poly Pomona service. Exercise caution when clicking links or opening attachments. Please forward suspicious email to suspectem...@cpp.edu<mailto:suspectem...@cpp.edu>.
