Hi Jeremy, All, We've finished the "Phase 1" of this Project, and here are some samples of the result: https://bit.ly/37ELoj1 (in 2 formats: single TW5 file & "*.tid" files) Also I created a video https://youtu.be/FuJR-5uSDVU trying to explain both the above results and the project itself.
P.S., 1- The Project Name has been changed to "510K" after creating the above video, cause I'm really interesting to see how TW5 will handle that number of tiddlers 2- "Phase 1" is work only with GG Conversations, Messages will be in "Phase 2" Thanks On Sunday, August 1, 2021 at 5:59:52 PM UTC Taacees ME wrote: > Hi Jeremy, > > I hope the following will enplane it better, but first of all I don't know > yet if this violate any licences or not and I'm still looking if any. > > Some Facts:- > 1- TiddlyWiki has multiple "Data Sources" (Communities) around the > internet (this GG, TW5 Dev GG, TW5 Doc GG, GitHub Discussions, Stack > Overflow Questions, TW5 on Reddit, ...... etc) > 2- TiddlyWiki is the BEST personal Information Management System we > know/used (this is a FACT for me, but to prove it to others, at lease TW > should manage its own information). > > Now, in a few words; my goal is to EXPORT all TiddlyWiki data/info from > those data sourced (by scraping/crawling) and IMPORT them into a unified > TiddlyWiki System (converting data units into tiddlers) so we can start > use/build over it. > In other words, let TiddlyWiki own its data that scattered around the > Internet and build a "TW Information Portal" that show the EXTREME power of > TW5 over its own data. > NOTE that this will NOT replace any of the original Data Sources, just to > complements them and construct a "Portal" that we can build over it > > So, at the end of the 1st phase , we'll have a single TiddlyWiki system > (Node.js) contains something like the following (ALL the below data will be > extracted from the source, NO Human Intervention at this phase): > > Imaginary TW5 Google Group Tiddler (we'll have > 120K of them) > =================================== > title: "GG TW5 ID pDlJDdWZNHQ" > tags: [[2021]] [[Conversation]] [[Message]] [[TiddlyWiki Google Group]] > custm-field-gg-title: "The History Show of the GG Community" > custm-field-gg-url: "https://groups.google.com/g/tiddlywiki/c/pDlJDdWZNHQ > " > custm-field-gg-author: "Taacees" > custm-field-gg-date: "01 Aug 2021" > text: "Message Body" > ..... Any other info in separated "Custom Fields" > AND a separated tiddler for each reply with links to main Question > > Imaginary GitHub Disscussion Tiddler > ===================================== > title: "GitHub Disscussions ID 5924" > tags: [[2021]] [[Disscussion]] [[TiddlyWiki GitHub Disscussions]] > custm-field-gethub-title: "Bitmap editor - should we use pointer events?" > custm-field-gethub-url: " > https://github.com/Jermolene/TiddlyWiki5/discussions/5924"; > custm-field-gethub-author: "BurningTreeC" > custm-field-gethub-date: "01 Aug 2021" > text: "Message Body" > ..... Any other info in separated "Custom Fields" > AND a separated tiddler for each reply with links to main Question > > Imaginary Reddit Quesion/Comment Tiddler > ========================================= > title: "Reddit Question ID onx6qn" > tages: [[2021]] [[Reddit]] [[Question]] > custm-field-reddit-title: "Newbie Question: Editing a field in a template" > custm-field-reddit-url: " > https://www.reddit.com/r/TiddlyWiki5/comments/onx6qn/newbie_question_editing_a_field_in_a_template/ > " > custm-field-reddit-author: "u/OneDiscombobulated83" > custm-field-reddit-date: "01 Aug 2021" > text: "Question Body" > ..... Any other info in separated "Custom Fields" > AND a separated tiddler for each reply with links to main Question > > Imaginary Stack Overflow Quesion/Answer Tiddler > =============================================== > title: "StackOverflow Answer ID 34693482" > tages: [[2016]] [[Stack Overflow]] [[Reply]] [[Answer]] > custom-field-stackoverflow-title: tiddlywiki: "can't save changes in > QWebView" > custom-field-stackoverflow-date: "9 Jan 2016" > custom-field-stackoverflow-url: " > https://stackoverflow.com/questions/34693482/tiddlywiki-cant-save-changes-in-qwebview > " > text: "Question Body" > ..... All other info in separated "Custom Fields" > AND separated tiddler for each answers with links to main Question > > I hope this make the idea more clear, but It'll be more visible after > showing some example "tid" files. > > And, regarding the Tools, I'm developing those scrapers/crawlers using the > following golang libraries: > > - gocolly/colly <https://github.com/gocolly/colly> > - chromedp/chromedp <https://github.com/chromedp/chromedp> > > Regards > > On Sunday, August 1, 2021 at 2:41:26 PM UTC jeremy...@gmail.com wrote: > >> Hi Taacees >> >> Thank you! The project looks intriguing, could you explain a little more? >> From the GitHub repo I understand that you’re trying to mine the Google >> Group archive for useful information, but I’d like to understand more about >> your goals and the techniques you are using, >> >> Best wishes >> >> Jeremy. >> >> >> On 1 Aug 2021, at 04:27, Taacees ME <taace...@gmail.com> wrote: >> >> Hi Everyone, >> >> Following the Jeremy's advice >> <https://twitter.com/Jermolene/status/1420658644211978240>, Kindly >> accept the invitation to celebrate that our GG community has reached the >> count of 25K conversations (almost 😉). >> Get your popcorn, soda, favorite audio tracks and enjoy the 16 Years age >> of our Community (try to forget any cons of the "Google Group" 😉) >> >> The History Show of TiddlyWiki GG Community >> <https://www.youtube.com/playlist?list=PLh3C3gr79VC1UxmXoGtaTyU1iPaw_WJg_> >> >> Regards >> >> -- >> You received this message because you are subscribed to the Google Groups >> "TiddlyWiki" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to tiddlywiki+...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/tiddlywiki/11512e17-2eab-48b5-b7a5-6ce46681e6bcn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/tiddlywiki/11512e17-2eab-48b5-b7a5-6ce46681e6bcn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f99d51b0-f004-40a7-9cbd-cee1343c56f5n%40googlegroups.com.
