Hi Mua, I found some useful function of Dataflow to import JSON type file from datastore entities, I'm struggling with this now. All looks good, but exported file does not include any data at all. https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#datastoretogcstext
Cheers, Han On Tuesday, 27 August 2019 13:52:35 UTC+12, Han Ju Kim wrote: > > Great! I will have a look at. > Still confusing, why google does not provide CSV,TXT or JSON export of > datastore. > > Cheers, > Han > > On Tuesday, 27 August 2019 13:26:18 UTC+12, Mua Rachmann wrote: >> >> If that sucks, You might want to have a look at this >> https://github.com/junosuarez/node-level-export >> >> Best, >> Mua >> >> On Tue, Aug 27, 2019 at 2:20 AM Mua Rachmann <[email protected]> wrote: >> >>> Okay, I think you might want to look at >>> https://github.com/nisaacson/leveldb-json/issues/1 which is pretty >>> similar to your logs, >>> Also you might want to build from source and consider a fork of the >>> original repo - https://github.com/killix/leveldb-json >>> which seems to solve the issue. Dependencies not being upgraded or so. - >>> Basically applying - >>> https://github.com/killix/leveldb-json/commit/35df5f3d396907e015527984aa03e13726e9e386 >>> >>> Can you try that and see if it solves your problem >>> >>> Regards, >>> Mua >>> >>> On Tue, Aug 27, 2019 at 2:13 AM Han Ju Kim <[email protected]> wrote: >>> >>>> This is my log. When I tried to "npm install -g leveldb-json", it comes >>>> up >>>> >>>> C:\WINDOWS\system32>npm install -g leveldb-json >>>> C:\Users\hkim\AppData\Roaming\npm\leveldb-load-from-json -> >>>> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\bin\load-from-json >>>> C:\Users\hkim\AppData\Roaming\npm\leveldb-dump-to-json -> >>>> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\bin\dump-to-json >>>> >>>> > [email protected] install >>>> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\node_modules\leveldown >>>> > node-gyp rebuild >>>> >>>> >>>> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\node_modules\leveldown>if >>>> >>>> not defined npm_config_node_gyp (node "C:\Program >>>> Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" >>>> >>>> rebuild ) else (node "C:\Program >>>> Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" >>>> rebuild ) >>>> gyp ERR! configure error >>>> gyp ERR! stack Error: Command failed: C:\Python37\python.EXE -c import >>>> sys; print "%s.%s.%s" % sys.version_info[:3]; >>>> gyp ERR! stack File "<string>", line 1 >>>> gyp ERR! stack import sys; print "%s.%s.%s" % sys.version_info[:3]; >>>> gyp ERR! stack ^ >>>> gyp ERR! stack SyntaxError: invalid syntax >>>> gyp ERR! stack >>>> gyp ERR! stack at ChildProcess.exithandler (child_process.js:294:12) >>>> gyp ERR! stack at ChildProcess.emit (events.js:198:13) >>>> gyp ERR! stack at maybeClose (internal/child_process.js:982:16) >>>> gyp ERR! stack at Process.ChildProcess._handle.onexit >>>> (internal/child_process.js:259:5) >>>> gyp ERR! System Windows_NT 10.0.18362 >>>> gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program >>>> Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" >>>> >>>> "rebuild" >>>> gyp ERR! cwd >>>> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\node_modules\leveldown >>>> gyp ERR! node -v v10.16.3 >>>> gyp ERR! node-gyp -v v3.8.0 >>>> gyp ERR! not ok >>>> npm ERR! code ELIFECYCLE >>>> npm ERR! errno 1 >>>> npm ERR! [email protected] install: `node-gyp rebuild` >>>> npm ERR! Exit status 1 >>>> npm ERR! >>>> npm ERR! Failed at the [email protected] install script. >>>> npm ERR! This is probably not a problem with npm. There is likely >>>> additional logging output above. >>>> >>>> npm ERR! A complete log of this run can be found in: >>>> npm ERR! >>>> >>>> C:\Users\hkim\AppData\Roaming\npm-cache\_logs\2019-08-27T01_11_59_731Z-debug.log >>>> >>>> C:\WINDOWS\system32> >>>> >>>> On Tuesday, 27 August 2019 13:10:37 UTC+12, Mua Rachmann wrote: >>>>> >>>>> Hi Kim, >>>>> >>>>> Can we see the logs here? >>>>> >>>>> On Tue, Aug 27, 2019 at 2:05 AM Han Ju Kim <[email protected]> wrote: >>>>> >>>>>> Hi Mua, >>>>>> >>>>>> I appreciate prompt feedback! I already tried that, but it showed >>>>>> some error and not updated quite long time period. Any help will be >>>>>> appreciated. >>>>>> >>>>>> Cheers, >>>>>> Han >>>>>> >>>>>> On Tuesday, 27 August 2019 11:50:47 UTC+12, Mua Rachmann wrote: >>>>>>> >>>>>>> Hi Kim, >>>>>>> >>>>>>> You might want to have a look at this - >>>>>>> https://github.com/nisaacson/leveldb-json >>>>>>> >>>>>>> regards. >>>>>>> Mua >>>>>>> >>>>>>> On Tue, Aug 27, 2019 at 12:35 AM Han Ju Kim <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I'm trying to export Datastore entities as JSON format. By using >>>>>>>> API or gsutil such as "gcloud datastore export", it only export as >>>>>>>> LevelDB >>>>>>>> format with many output-xxx files. Is there any tool to converting >>>>>>>> this >>>>>>>> LevelDB files to JSON format or any third party engine to do that? >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Google App Engine" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/google-appengine/06c074cc-cc4a-4c73-a8d9-ed7299d5421e%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/google-appengine/06c074cc-cc4a-4c73-a8d9-ed7299d5421e%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/google-appengine/22e67aef-22ad-4dcd-9fdd-ecaaa89539bc%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/google-appengine/22e67aef-22ad-4dcd-9fdd-ecaaa89539bc%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/google-appengine/7cd042ec-6ecc-4a16-9aba-889606d02191%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/google-appengine/7cd042ec-6ecc-4a16-9aba-889606d02191%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bc5d485b-6366-43e3-9b95-0b2516658061%40googlegroups.com.
