[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney updated NUTCH-1481: ---------------------------------------- Fix Version/s: 2.2 > When using MySQL as storage unicode characters within URLS cause nutch to fail > ------------------------------------------------------------------------------ > > Key: NUTCH-1481 > URL: https://issues.apache.org/jira/browse/NUTCH-1481 > Project: Nutch > Issue Type: Bug > Components: crawldb > Affects Versions: 2.1 > Environment: mysql 5.5.28 on centos > Reporter: Arni Sumarlidason > Labels: database, sql, unicode, utf8 > Fix For: 2.2 > > > MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. > currently the url of a web page is used as a primary key in nutch storage. > when using latin1 character set on the 'id' column @ length 767 > bytes/characters; unicode characters in urls cause jdbc to throw an exception, > java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: > '\xE2\x80\x8' for column 'id' at row 1 > when using utf8mb4 character set on the 'id' column @ length 190 characters / > 760 bytes to fully support unicode characters; the field length becomes > insufficient > It may be better to use a hash of the url as the primary key instead of the > url itself. This would allow urls of any length and full utf8 support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira