Thanks for the reply Jason, Yea the ticket is currently in Heroku Support - I’m not sure how helpful they are - haven’t heard a response yet. I actually tested this against my clients production app (it’s paid) so there can’t be anything wrong with the service itself terms of a 10,000 rows limit I’ve received before.
Can you explain more about why the runner would have “max memory on it’s ‘headless’ runner dyno” It can’t be a memory issue because I tested the upload with only one row. I haven’t tried a rake task, but would it be any different? Gary > On Apr 2, 2015, at 5:30 AM, Jason Fleetwood-Boldt <[email protected]> > wrote: > > > Gary-- > > Hmm.... definitely something in the runner. My guess is that the runner has > a max memory on its "headless" runner dyno and you've hit against the max > memory by trying to load a CSV file. > > Did you try this using a rake task instead? > > What does Heroku support say? They should be able to tell you what is going > on there. > > -Jason > > > >> On Apr 1, 2015, at 4:16 PM, Gary Tsai <[email protected] >> <mailto:[email protected]>> wrote: >> >> I'm having issues importing a CSV using a rails runner command once my >> application code is deployed to Heroku. >> >> The CSV gets started from the command line like this: >> >> heroku run rails runner scripts/import_csv.rb < lesley_grades.csv >> which then gets initialized here >> >> CSVImporter.new.import!($stdin) >> This is the code for the CVS import >> >> def import!(csv) >> p "Before database transaction" >> ActiveRecord::Base.transaction do >> p "I'm here before CSV.parse!!!!!" >> csv_reader = CSV.parse(csv, row_sep: "\r") do |row| >> p header_row?(row) >> p "Header row!?" >> set_record_class_and_columns(row) if header_row?(row) >> >> if columns_mapping_defined? && record_class_defined? && >> record_row?(row) >> import_row(row) >> end >> end >> if imports_failed? >> puts 'Aborting importing and rolling back...' >> show_errors >> raise ActiveRecord::Rollback >> end >> end >> end >> Here is the log and the error message that gets printed after I control+c >> from the stall. >> >> heroku run rails runner scripts/import_csv.rb < >> ~/Desktop/utc_csv_export/lesley_grades.csv >> Running `rails runner scripts/import_csv.rb` attached to terminal... up, >> run.9179 >> LesleyGrade,lesley_id,last,first,active,site,cohort,section,sections_title,faculty,completed_term_cred,term,sec_start_date,sec_end_date,grade,stc_cred,active_program,most_recent_program,intent_filed,stc_term_gpa,sta_cum_gpa,start_term,prog_status,last_change_date >> ,1654038,John,Doe,TRUE,"Baltimore, MD",2012,14/FA_ERLIT_6999_U15AA,Directed >> Independent Study,"McMackin Mary , Richardson >> Kathleen",2,14/FA,9/3/14,12/17/14,S,2,EME.2270.TCBAL.01,EME.2270.TCBAL.01, >> ,3.3,3.148,12/SU,A,9/2/14** [NewRelic][03/31/15 18:30:10 +0000 >> faf23bac-4be0-4098-82e5-612003c543e6 (3)] INFO : Starting the New Relic >> agent in "staging" environment. >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : To prevent agent startup add a NEWRELIC_AGENT_ENABLED=false >> environment variable or modify the "staging" section of your newrelic.yml. >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Reading configuration from config/newrelic.yml >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Environment: staging >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : No known dispatcher detected. >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Application: My Application (Staging) >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing deferred Rack instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing Rack::Builder middleware instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing Net instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing Rails 3+ middleware instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing middleware-based Excon instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing ActiveRecord 4 instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing Rails 4 Controller instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing Rails 4 Error instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Installing Rails 4 view instrumentation >> ** [NewRelic][03/31/15 18:30:10 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Finished instrumentation >> ** [NewRelic][03/31/15 18:30:11 +0000 faf23bac-4be0-4098-82e5-612003c543e6 >> (3)] INFO : Reporting to: >> https://rpm.newrelic.com/accounts/773165/applications/4986263 >> "Before database transaction" >> "I'm here before CSV.parse!!!!!" >> ^C^C** [NewRelic][03/31/15 18:35:15 +0000 >> faf23bac-4be0-4098-82e5-612003c543e6 (3)] INFO : Starting Agent shutdown >> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/csv.rb:1778:in `gets': Interrupt >> from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/csv.rb:1778:in `block in >> shift' >> from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/csv.rb:1776:in `loop' >> from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/csv.rb:1776:in `shift' >> from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/csv.rb:1718:in `each' >> from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/csv.rb:1296:in `parse' >> from /app/lib/import/csv_importer.rb:14:in `block in import!' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in >> `block in transaction' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:219:in >> `within_new_transaction' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in >> `transaction' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/transactions.rb:208:in >> `transaction' >> from /app/lib/import/csv_importer.rb:12:in `import!' >> from scripts/import_csv.rb:3:in `<top (required)>' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands/runner.rb:60:in >> `load' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands/runner.rb:60:in >> `<top (required)>' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands/commands_tasks.rb:128:in >> `require' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands/commands_tasks.rb:128:in >> `require_command!' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands/commands_tasks.rb:95:in >> `runner' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands/commands_tasks.rb:40:in >> `run_command!' >> from >> /app/vendor/bundle/ruby/2.0.0/gems/railties-4.1.1/lib/rails/commands.rb:17:in >> `<top (required)>' >> from /app/bin/rails:4:in `require' >> from /app/bin/rails:4:in `<main>' >> As you can see from the console I can't get passed this line(14): >> >> csv_reader = CSV.parse(csv, row_sep: "\r") >> Is this a Dyno's issue or even a New Relic issue? So far I'm running this on >> a free heroku instance, and I only uploaded 2 rows just to test it, and it >> still doesn't work. >> >> Anyone have any clues? Thanks! >> >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Heroku" group. >> >> To unsubscribe from this group, send email to >> [email protected] >> <mailto:[email protected]> >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en_US?hl=en >> <http://groups.google.com/group/heroku?hl=en_US?hl=en> >> >> --- >> You received this message because you are subscribed to the Google Groups >> "Heroku Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > ---- > > Jason Fleetwood-Boldt > [email protected] <mailto:[email protected]> > http://www.jasonfleetwoodboldt.com/writing > <http://www.jasonfleetwoodboldt.com/writing> > > -- > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > > To unsubscribe from this group, send email to > [email protected] > <mailto:[email protected]> > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en_US?hl=en > <http://groups.google.com/group/heroku?hl=en_US?hl=en> > > --- > You received this message because you are subscribed to a topic in the Google > Groups "Heroku Community" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/heroku/UNhif0CoY8Q/unsubscribe > <https://groups.google.com/d/topic/heroku/UNhif0CoY8Q/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- -- You received this message because you are subscribed to the Google Groups "Heroku" group. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/heroku?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "Heroku Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
